Building robust Python packages often involves more than just source code. Many applications rely on additional resources like configuration files, static assets (images, CSS, JavaScript), templates, or data files. Successfully distributing these non-Python files alongside your code is a critical aspect of creating a functional and user-friendly package. This guide will walk you through how to include package data with setuptools/distutils, ensuring your distributed applications work seamlessly out-of-the-box. We’ll explore the primary mechanisms provided by the Python packaging ecosystem, offering practical examples and best practices to help you manage your project’s assets effectively.
Understanding Package Data in Python Projects
Package data refers to any non-code files that a Python package needs to function correctly after installation. Think of them as ancillary resources essential for the application’s runtime. Examples include database schema files, UI layouts, documentation, or even pre-trained machine learning models. Without these assets properly included, your Python application might fail to load resources, display correctly, or perform its intended operations, leading to a broken user experience.
The Python packaging tools, primarily Setuptools (which largely supersedes the older Distutils for modern projects), provide several mechanisms to ensure these critical resource files are bundled with your source code during the distribution process. Understanding these methods is key to creating self-contained, easily installable packages. Failing to properly manage your package data can lead to frustrating installation issues and runtime errors for your users, undermining the reliability of your software.
From a developer’s perspective, correctly bundling package data simplifies deployment and ensures consistency across different environments. It means that once a user installs your package, all necessary components are available in their expected locations, regardless of their system configuration. This attention to detail is a hallmark of professional Python package development, enhancing both the usability and maintainability of your projects.
The package_data Option: Your First Approach
The package_data option in your setup.py file is the most common and straightforward way to include data files that reside inside your Python package directories. This method is ideal for resources that are tightly coupled with specific modules, such as a template file for a web framework component or an icon for a GUI application. It specifies patterns for files that should be copied into the installation location of the package itself.
To utilize package_data, you provide a dictionary mapping package names to a list of glob patterns for the files you want to include. For instance, if you have a package named my_app and it contains a templates directory with HTML files and a static directory with images, your setup.py might look something like this:
from setuptools import setup, find_packages setup( name='my_app', version='0.1.0', packages=find_packages(), package_data={ 'my_app': ['templates/.html', 'static/images/.png'], 'my_app.subpackage': ['data/.json'] Example for a subpackage }, ... other options )
This approach ensures that when my_app is installed, all specified HTML and PNG files will be placed alongside your Python modules. It’s crucial that the paths specified in package_data are relative to the package directory itself. This direct control over file inclusion within your package structure makes package_data highly effective for tightly integrated resources.
- Pros: Explicit, easy to understand for simple cases, and directly ties data files to their respective packages.
- Cons: Can become verbose for many files or deeply nested directories, and requires manual updates if new file types or locations are added.
Leveraging include_package_data and MANIFEST.in
While package_data offers explicit control, for larger projects with numerous resource files or dynamic file structures, managing all paths directly in setup.py can become cumbersome. This is where include_package_data=True in conjunction with a MANIFEST.in file becomes invaluable. This powerful combination automates the inclusion of package data based on a set of rules defined in MANIFEST.in.
When you set include_package_data=True in your setup.py file, Setuptools will look for a MANIFEST.in file in your project’s root directory. This file acts as a manifest template, dictating which non-Python files (and directories) from your source distribution should be included in the final package. It’s particularly useful for project structures where resource files might be spread across various subdirectories within your main package.
A typical MANIFEST.in file uses simple directives like include, exclude, recursive-include, and recursive-exclude. For example, to include all .txt files in a data directory within my_app and all .css files in a static directory, your MANIFEST.in might look like this:
recursive-include my_app/data .txt recursive-include my_app/static .css
This method is highly scalable and keeps your setup.py cleaner, as the file inclusion logic is externalized. It’s the recommended approach by the Python Packaging Authority (PyPA) for most projects needing to include package data efficiently. Remember that MANIFEST.in paths are relative to the project root, not the package root.
- Key Directives for MANIFEST.in:
include FILE_PATTERN: Includes files matching the pattern from the project root.exclude FILE_PATTERN: Excludes files matching the pattern from the project root.recursive-include DIR_PATTERN FILE_PATTERN: Recursively includes files matching the pattern within directories matchingDIR_PATTERN.recursive-exclude DIR_PATTERN FILE_PATTERN: Recursively excludes files matching the pattern within directories matchingDIR_PATTERN.graft DIR: Includes all files in the specified directory and its subdirectories.prune DIR: Excludes all files in the specified directory and its subdirectories.
The data_files Option: For Files Outside Python Packages
While package_data and MANIFEST.in handle files inside your Python packages, sometimes you need to install data files into system-wide locations or directories that are not part of your Python package structure. This is where the data_files option comes into play. It’s particularly useful for installing configuration files, documentation, or executable scripts that should reside in specific, non-Python-package-related directories on the target system.
The data_files option in setup.py takes a list of (destination_directory, [source_files]) tuples. The destination_directory is typically relative to the installation prefix (e.g., /usr/local on Linux), and the source_files is a list of paths to the files you want to install, relative to your setup.py file. For instance, you might Question & Answer :
When using setuptools, I can not get the installer to pull in any package_data files. Everything I’ve read says that the following is the correct way to do it. Can someone please advise?
setup( name='myapp', packages=find_packages(), package_data={ 'myapp': ['data/*.txt'], }, include_package_data=True, zip_safe=False, install_requires=['distribute'], )
where myapp/data/ is the location of the data files.
I realize that this is an old question, but for people finding their way here via Google: package_data is a low-down, dirty lie. It is only used when building binary packages (python setup.py bdist ...) but not when building source packages (python setup.py sdist ...). This is, of course, ridiculous – one would expect that building a source distribution would result in a collection of files that could be sent to someone else to built the binary distribution.
In any case, using MANIFEST.in will work both for binary and for source distributions.
