APE 17 Migration Guide¶
The Astropy project is now transitioning from using astropy-helpers for infrastructure to more standard Python packaging tools. The motivation and implications of this are discussed in an Astropy Proposal for Enhancements: APE 17: A roadmap for package infrastructure without astropy-helpers. The core astropy package has already migrated and these changes will be included in astropy v4.1.
This page aims to provide a guide to migrating to using the new infrastructure described in APE 17. We assume that your package is currently using astropy-helpers and that you have used a version of the Astropy package-template in the past to set up your package (though this guide might still be useful if the latter is not the case).
Throughout this guide, we will assume that your package is called my-package
and that the module is called my_package
. We deliberately choose a name
where the package name is different from the module name, but for many cases,
these will be the same.
Step 0: Re-rendering the template¶
In this guide, we will describe the changes to make to the files in your package. To make some of the steps easier below, you should first re-run cookiecutter to re-render the template to a temporary folder:
cookiecutter gh:astropy/package-template -o my_package_tmp
We will refer to some of the rendered files in several of the steps below.
Step 1: Remove astropy-helpers¶
To remove the astropy-helpers submodule, first, run:
git rm -r astropy_helpers
if astropy_helpers
was the only submodule in your repository, the
.gitmodules
file will be empty, you can remove this if you wish:
git rm -f .gitmodules
Next you should remove the ah_bootstrap.py
file:
git rm ah_bootstrap.py
You can now commit your changes with:
git commit -m "Remove astropy-helpers submodule"
Step 2: Update/create setup.cfg
¶
The next step is to update make sure that you have a setup.cfg
file that contains meta-data about the package. If you already have this file,
you will likely need to update it, and if you don’t already have this file, you
will need to create it by copying over the setup.cfg
file generated in
Step 0. If you are updating an existing file, you may
also find it easier to start from the file generated in Step 0
and add back any customizations.
In any case this file should contain at least the following entries:
[metadata]
name = my-package
author = ...
author_email = ...
license = ...
license_file = LICENSE.rst
url = ...
description = ...
long_description = file: README.rst
[options]
zip_safe = False
packages = find:
install_requires =
numpy
astropy
...
python_requires = >=3.6
Replace the ...
with the information for your package. Make sure
that license_file
and long_description
have the right filename,
and specify your required dependencies one per line in the
install_requires
section. Make sure the python_requires
line is
set to indicate the correct minimum Python version for your package.
If you already had a file, make sure you remove the following entries (if present):
package_name
version
tests_require
[ah_bootstrap]
and all entries in it
Also remove any existing setup_requires
entry, though we’ll add
back this key for the setuptools-scm package in
Step 8.
Step 3 - Define optional, test, and docs dependencies¶
Next up, add a new section to the setup.cfg
file (or modify, if it
already exists), to specify optional dependencies as well as
dependencies required to run tests and build the documentation, for
example:
[options.extras_require]
all =
scipy
matplotlib
test =
pytest-astropy
docs =
sphinx-astropy
If you don’t need any optional dependencies, remove the all
section.
You will likely need to have at least pytest-astropy
in the test
section and sphinx-astropy
in docs
.
Step 4 - Define package data¶
If your package includes non-Python data files, you will need to update
how you declare which data files to include. If you have been using the
Astropy package template, it is likely that you have functions called
get_package_data
defined inside setup_package.py
files. Remove
these functions, and instead define the package data using a
[options.package_data]
section inside your setup.cfg
file,
e.g.:
[options.package_data]
* = *.fits, *.csv
my_package.tests = data/*
In the above example, all .fits
and .csv
in the package will be
included as well as all files inside my_package/tests/data
.
Step 5 - Update your setup.py
file¶
Copy the setup.py
file you generated in Step 0 and replace your existing one
- it should be good to go as-is without any further customizations.
Step 6: add a pyproject.toml
file¶
The pyproject.toml
file is used to declare dependencies needed to run
setup.py
and build the package. Copy the pyproject.toml
file you
generated in Step 0 and replace your existing one.
If your package doesn’t have any compiled extensions, the file should contain:
[build-system]
requires = ["setuptools",
"setuptools_scm",
"wheel"]
build-backend = 'setuptools.build_meta'
Step 7 - Handling C/Cython extensions¶
If your package has no compiled C/Cython extensions, you can skip this
step. Otherwise, if you have C or Cython extensions, you can either
define your extensions manually inside the setup.py
file or make use
of the extension-helpers
package to collect extensions in a similar way to astropy-helpers.
Step 7a - Defining extensions manually¶
You can define extensions manually as described
here.
If you do this, you can remove all setup_package.py
files in your
package, and you don’t need to include extension-helpers in the
pyproject.toml
file.
If you have Cython extensions or your extensions use the NumPy C API, proceed to Step 7c, otherwise you can proceed to Step 8.
Step 7b - Using extension-helpers¶
You can use the extension-helpers package to:
- Automatically define extensions for Cython files.
- Pick up extensions declared in
setup_package.py
files, as described in the extension-helpers documentation.
The latter works by looking through all the setup_package.py
files
in your package and executing the get_extensions()
functions, which
each should return a list of extensions. Check through your existing
setup_package.py
files (if any), and make sure that any
astropy_helpers
imports are changed to extension_helpers
. Also
note that all functions in extension-helpers should now be imported from
the top level. See the extension-helpers API documentation for a complete
list of functions still provided by extension-helpers. Finally, make
sure that any instance of include_dirs='numpy'
is changed to
include_dirs=np.get_include()
and add the import numpy as np
import if not already present.
Note that if you have existing setup_package.py
files from an older version
of the package template, you may have imports that look like:
from astropy_helpers import setup_helpers
and extensions that make use of:
cfg = setup_helpers.DistutilsExtensionArgs()
You should replace the import, from astropy_helpers import setup_helpers
,
with:
from collections import defaultdict
and change any lines that use setup_helpers.DistutilsExtensionArgs()
to
instead use:
cfg = defaultdict(list)
Provided you indicated when you generated the template in Step 0 that you wanted to use compiled extensions, you should be good to go. If not, make sure you add:
from extension_helpers.setup_helpers import get_extensions
just under the following lines at the top of the setup.py
file:
import sys
from setuptools import setup
In addition, in the same file, add ext_modules=get_extensions()
to the
call to setup.py
.
If you have Cython extensions or your extensions use the NumPy C API, proceed to Step 7c, otherwise you can proceed to Step 8.
Step 7c - Cython and Numpy build-time dependencies¶
If your compiled extensions rely on the NumPy C API, you will need to
declare Numpy as a build-time dependency in pyproject.toml
. Note
that as described in APE
17,
you need to pin the build-time Numpy dependency to the oldest
supported Numpy version for each Python version. However, rather than doing this
manually, you can add the oldest-supported-numpy
package to the build
dependencies in your pyproject.toml
file. In addition if you have Cython
extensions, you will need to also add an entry for Cython, pinning it to a
recent version. Provided you indicated when you generated the template in Step 0
that you wanted to use compiled extensions, you should be good to go as both
oldest-supported-numpy
and cython
should be in the pyproject.toml
file. In this case your pyproject.toml
file will look like:
[build-system]
requires = ["setuptools",
"setuptools_scm",
"wheel",
"extension-helpers",
"oldest-supported-numpy",
"cython==0.29.14"]
build-backend = 'setuptools.build_meta'
Whenever a new major Python version is released, you will likely need to update the Cython pinning to use the most recent Cython version available.
Step 8 - Using setuptools_scm¶
The setuptools_scm
package is now recommended to manage the version numbers for your
package. The way this works is that instead of setting the version
number manually in, e.g., setup.cfg
or elsewhere in your package,
the version number is based on git tags.
In Steps 5 and 6, we already
added the required entry for setuptools_scm to setup.py
and
pyproject.toml
.
In addition to these, we recommend that you define setup_requires
inside the
[options]
section of your setup.cfg
file:
[options]
...
setup_requires = setuptools_scm
...
This will already be the case if you copied the setup.cfg
generated in
Step 0. Having setup_requires
is not strictly
necessary but will make it possible for python setup.py --version
to work
without having to install setuptools_scm
manually.
Next, check your .gitignore
and make sure that you have a line containing:
my_package/version.py
Finally, copy over the _astropy_init.py
file generated in Step 0, or
alternatively edit your my_package/_astropy_init.py
file and remove the
following lines:
try:
from .version import githash as __githash__
except ImportError:
__githash__ = ''
and remove '__githash__'
from the __all__
list at the top of the
file. The git hash is now contained in the version number, so this is no
longer needed.
Step 9 - Configuring pytest¶
To make sure that pytest works properly, you can set a few options in a
[tool:pytest]
section in your setup.cfg
file:
[tool:pytest]
testpaths = "my_package" "docs"
astropy_header = true
doctest_plus = enabled
text_file_format = rst
addopts = --doctest-rst
For the testpaths
line, make sure you replace my_package
with the name
of your package. This section will already exist if you copied the setup.cfg
generated in Step 0.
The remaining options ensure that the output from pytest includes a
header that lists dependencies and system information, and also ensure
that the .rst
files are picked up and tested by pytest.
Step 10 - Update MANIFEST.in
¶
Edit your MANIFEST.in
file to remove the following lines, if present
(and any other line related to astropy_helpers
) - those lines might
include any of the following:
include ez_setup.py
include ah_bootstrap.py
# the next few stanzas are for astropy_helpers. It's derived from the
# astropy_helpers/MANIFEST.in, but requires additional includes for the actual
# package directory and egg-info.
include astropy_helpers/README.rst
include astropy_helpers/CHANGES.rst
include astropy_helpers/LICENSE.rst
recursive-include astropy_helpers/licenses *
include astropy_helpers/ez_setup.py
include astropy_helpers/ah_bootstrap.py
recursive-include astropy_helpers/astropy_helpers *.py *.pyx *.c *.h
recursive-include astropy_helpers/astropy_helpers.egg-info *
# include the sphinx stuff with "*" because there are css/html/rst/etc.
recursive-include astropy_helpers/astropy_helpers/sphinx *
prune astropy_helpers/build
prune astropy_helpers/astropy_helpers/tests
Then add a new line near the top with the following:
include pyproject.toml
Step 11 - Updating your documentation configuration¶
You will need to edit the docs/conf.py
file to make sure it does not
use astropy-helpers. If you see a code block such as:
try:
import astropy_helpers
except ImportError:
# Building from inside the docs/ directory?
if os.path.basename(os.getcwd()) == 'docs':
a_h_path = os.path.abspath(os.path.join('..', 'astropy_helpers'))
if os.path.isdir(a_h_path):
sys.path.insert(1, a_h_path)
# Load all of the global Astropy configuration
from astropy_helpers.sphinx.conf import *
# Get configuration information from setup.cfg
try:
from ConfigParser import ConfigParser
except ImportError:
from configparser import ConfigParser
you should change it to:
try:
from sphinx_astropy.conf.v1 import * # noqa
except ImportError:
print('ERROR: the documentation requires the sphinx-astropy package to be installed')
sys.exit(1)
# Get configuration information from setup.cfg
from configparser import ConfigParser
conf = ConfigParser()
Find and replace any instances of package_name
in the file with
name
.
Step 12 - Setting up tox¶
tox is a tool for automating commands, which is well suited to, e.g., running tests for your package or building the documentation. One of the benefits of using tox is that it will (by default) create a source distribution for your package and install it into a virtual environment before running tests or building docs, which means that it will be a good test of whether, e.g., you have declared the package data correctly.
As a starting point, copy over the tox.ini
file generated in Step 0.
You can always then customize it if needed (although it should work out
of the box).
Once you have done this you should be able to do the following:
Run tests with minimal dependencies:
tox -e test
Run tests with astropy LTS and Numpy 1.16:
tox -e test-astropylts-numpy116
Run tests with all optional dependencies:
tox -e test-alldeps
Run tests with minimal dependencies and the latest developer version of numpy and astropy:
tox -e test-devdeps
Build the documentation:
tox -e build_docs
Run code style checks on your code:
tox -e codestyle
The {posargs}
corresponds to arguments passed to tox
after a
--
separator - for example to make pytest verbose in a test
environment, you can do:
tox -e test -- -v
Step 13 - Updating your Continuous Integration¶
This step will depend on what continuous integration services you use. Broadly speaking, unless there are dependencies you need that can only be installed with conda, you should no longer need to use ci-helpers to install these. The recommended approach is to use the tox file to set up the different configurations you want to use, and to then keep the CI configuration as simple as possible.
If you use Travis CI, a good place to start is the .travis.yml
file
generated in Step 0, and you can then see if any previous customizations you had
made need to be copied over. This file shows how one can configure Travis to use
tox, optionally using conda via ci-helpers to set up Python on MacOS X and
Windows.
Step 14 - Update ReadTheDocs configuration¶
With the set-up described in this migration guide, you should be able to
simplify the configuration for ReadTheDocs. This can be done via a
.readthedocs.yml
or readthedocs.yml
file (the former is recommended).
See the ReadTheDocs
documentation for more information about this file.
You should be able to copy over the .readthedocs.yml
file created in Step 0.
With this updated file, you should now be able to remove any pip requirements
file or conda YAML file that were previously used by readthedocs.yml
.
Step 15 - Coverage configuration¶
Preivously, astropy-helpers expected the coverage configuration to
be located in my_package/tests/coveragerc
. This is now no longer
necessary, so you can now define the coverage configuration inside
the setup.cfg
file, which should help reduce the number of files
to keep track of. Add the following to the bottom of your setup.cfg
:
[coverage:run]
omit =
my_package/_astropy_init*
my_package/conftest.py
my_package/*setup_package*
my_package/tests/*
my_package/*/tests/*
my_package/extern/*
my_package/version*
*/my_package/_astropy_init*
*/my_package/conftest.py
*/my_package/*setup_package*
*/my_package/tests/*
*/my_package/*/tests/*
*/my_package/extern/*
*/my_package/version*
[coverage:report]
exclude_lines =
# Have to re-enable the standard pragma
pragma: no cover
# Don't complain about packages we have installed
except ImportError
# Don't complain if tests don't hit assertions
raise AssertionError
raise NotImplementedError
# Don't complain about script hooks
def main\(.*\):
# Ignore branches that don't pertain to this version of Python
pragma: py{ignore_python_version}
# Don't complain about IPython completion helper
def _ipython_key_completions_
Make sure to replace my_package
by your module name. If you had any
customizations in coveragerc
you can include them here, but otherwise the
above should be sufficient and you should now be able to remove the old file:
git rm my_package/tests/coveragerc
Step 16 - conftest.py file updates¶
For the header in your test runs to be correct with the latest versions of
astropy, you will need to make sure that you update your conftest.py
file as
described in the pytest-astropy-header instructions.
You can also copy over the file created in Step 0 and add back any
customizations you had.
Step 17 - Final cleanup¶
Once you’ve made the above changes, you should be able to remove the
following sections from your setup.cfg
file:
[build_docs]
[build_sphinx]
[upload_docs]
You should also add pip-wheel-metadata/
to your .gitignore
file.
Once you are done, if you would like us to help by reviewing your changes, you can open a pull request to your package and mention @astrofrog or @Cadair to ask for a review.
Note on releasing your package¶
As a result of the changes above, there are some tweaks to the procedure to follow for releasing your package - see the latest instructions in the astropy documentation. The two main changes are that you no longer need to manually update the version number in files, instead the version number is based on the latest git tag, and in addition the source file should be built using the pep517 package.
Note on conda recipes and pyproject.toml¶
While not something you can do until you release your updated package, you will
need to take care to update conda recipes (e.g., in conda-forge) for your
package. In particular, since conda ignores pyproject.toml
files,
you will need to make sure that the build dependencies present in pyproject.toml
are explicitly listed as build dependencies in the conda recipe. For
packages with compiled extensions and Cython, this would look like:
build:
- {{ compiler('c') }}
host:
- pip
- python
- setuptools
- setuptools_scm
- numpy
- cython
- extension-helpers
run:
...
while for pure-Python packages you will still need to make sure
setuptools_scm
is included in the build dependencies:
host:
- pip
- python
- setuptools
- setuptools_scm
run:
...