pyrox.dev
1# Python {#python}
2
3## Reference {#reference}
4
5### Interpreters {#interpreters}
6
7| Package | Aliases | Interpreter |
8|------------|-----------------|-------------|
9| python27 | python2, python | CPython 2.7 |
10| python39 | | CPython 3.9 |
11| python310 | | CPython 3.10 |
12| python311 | python3 | CPython 3.11 |
13| python312 | | CPython 3.12 |
14| python313 | | CPython 3.13 |
15| pypy27 | pypy2, pypy | PyPy2.7 |
16| pypy39 | pypy3 | PyPy 3.9 |
17
18The Nix expressions for the interpreters can be found in
19`pkgs/development/interpreters/python`.
20
21All packages depending on any Python interpreter get appended
22`out/{python.sitePackages}` to `$PYTHONPATH` if such directory
23exists.
24
25#### Missing `tkinter` module standard library {#missing-tkinter-module-standard-library}
26
27To reduce closure size the `Tkinter`/`tkinter` is available as a separate package, `pythonPackages.tkinter`.
28
29#### Attributes on interpreters packages {#attributes-on-interpreters-packages}
30
31Each interpreter has the following attributes:
32
33- `libPrefix`. Name of the folder in `${python}/lib/` for corresponding interpreter.
34- `interpreter`. Alias for `${python}/bin/${executable}`.
35- `buildEnv`. Function to build python interpreter environments with extra packages bundled together. See [](#python.buildenv-function) for usage and documentation.
36- `withPackages`. Simpler interface to `buildEnv`. See [](#python.withpackages-function) for usage and documentation.
37- `sitePackages`. Alias for `lib/${libPrefix}/site-packages`.
38- `executable`. Name of the interpreter executable, e.g. `python3.10`.
39- `pkgs`. Set of Python packages for that specific interpreter. The package set can be modified by overriding the interpreter and passing `packageOverrides`.
40
41### Building packages and applications {#building-packages-and-applications}
42
43Python libraries and applications that use `setuptools` or
44`distutils` are typically built with respectively the [`buildPythonPackage`](#buildpythonpackage-function) and
45[`buildPythonApplication`](#buildpythonapplication-function) functions. These two functions also support installing a `wheel`.
46
47All Python packages reside in `pkgs/top-level/python-packages.nix` and all
48applications elsewhere. In case a package is used as both a library and an
49application, then the package should be in `pkgs/top-level/python-packages.nix`
50since only those packages are made available for all interpreter versions. The
51preferred location for library expressions is in
52`pkgs/development/python-modules`. It is important that these packages are
53called from `pkgs/top-level/python-packages.nix` and not elsewhere, to guarantee
54the right version of the package is built.
55
56Based on the packages defined in `pkgs/top-level/python-packages.nix` an
57attribute set is created for each available Python interpreter. The available
58sets are
59
60* `pkgs.python27Packages`
61* `pkgs.python3Packages`
62* `pkgs.python39Packages`
63* `pkgs.python310Packages`
64* `pkgs.python311Packages`
65* `pkgs.python312Packages`
66* `pkgs.python313Packages`
67* `pkgs.pypyPackages`
68
69and the aliases
70
71* `pkgs.python2Packages` pointing to `pkgs.python27Packages`
72* `pkgs.python3Packages` pointing to `pkgs.python311Packages`
73* `pkgs.pythonPackages` pointing to `pkgs.python2Packages`
74
75#### `buildPythonPackage` function {#buildpythonpackage-function}
76
77The `buildPythonPackage` function has its name binding in
78`pkgs/development/interpreters/python/python-packages-base.nix` and is
79implemented in `pkgs/development/interpreters/python/mk-python-derivation.nix`
80using setup hooks.
81
82The following is an example:
83
84```nix
85{ lib
86, buildPythonPackage
87, fetchPypi
88
89# build-system
90, setuptools-scm
91
92# dependencies
93, attrs
94, pluggy
95, py
96, setuptools
97, six
98
99# tests
100, hypothesis
101 }:
102
103buildPythonPackage rec {
104 pname = "pytest";
105 version = "3.3.1";
106 pyproject = true;
107
108 src = fetchPypi {
109 inherit pname version;
110 hash = "sha256-z4Q23FnYaVNG/NOrKW3kZCXsqwDWQJbOvnn7Ueyy65M=";
111 };
112
113 postPatch = ''
114 # don't test bash builtins
115 rm testing/test_argcomplete.py
116 '';
117
118 build-system = [
119 setuptools-scm
120 ];
121
122 dependencies = [
123 attrs
124 py
125 setuptools
126 six
127 pluggy
128 ];
129
130 nativeCheckInputs = [
131 hypothesis
132 ];
133
134 meta = {
135 changelog = "https://github.com/pytest-dev/pytest/releases/tag/${version}";
136 description = "Framework for writing tests";
137 homepage = "https://github.com/pytest-dev/pytest";
138 license = lib.licenses.mit;
139 maintainers = with lib.maintainers; [ domenkozar lovek323 madjar lsix ];
140 };
141}
142```
143
144The `buildPythonPackage` mainly does four things:
145
146* In the [`buildPhase`](#build-phase), it calls `${python.pythonOnBuildForHost.interpreter} setup.py bdist_wheel` to
147 build a wheel binary zipfile.
148* In the [`installPhase`](#ssec-install-phase), it installs the wheel file using `pip install *.whl`.
149* In the [`postFixup`](#var-stdenv-postFixup) phase, the `wrapPythonPrograms` bash function is called to
150 wrap all programs in the `$out/bin/*` directory to include `$PATH`
151 environment variable and add dependent libraries to script's `sys.path`.
152* In the [`installCheck`](#ssec-installCheck-phase) phase, `${python.interpreter} setup.py test` is run.
153
154By default tests are run because [`doCheck = true`](#var-stdenv-doCheck). Test dependencies, like
155e.g. the test runner, should be added to [`nativeCheckInputs`](#var-stdenv-nativeCheckInputs).
156
157By default `meta.platforms` is set to the same value
158as the interpreter unless overridden otherwise.
159
160##### `buildPythonPackage` parameters {#buildpythonpackage-parameters}
161
162All parameters from [`stdenv.mkDerivation`](#sec-using-stdenv) function are still supported. The
163following are specific to `buildPythonPackage`:
164
165* `catchConflicts ? true`: If `true`, abort package build if a package name
166 appears more than once in dependency tree. Default is `true`.
167* `disabled ? false`: If `true`, package is not built for the particular Python
168 interpreter version.
169* `dontWrapPythonPrograms ? false`: Skip wrapping of Python programs.
170* `permitUserSite ? false`: Skip setting the `PYTHONNOUSERSITE` environment
171 variable in wrapped programs.
172* `pyproject`: Whether the pyproject format should be used. When set to `true`,
173 `pypaBuildHook` will be used, and you can add the required build dependencies
174 from `build-system.requires` to `build-system`. Note that the pyproject
175 format falls back to using `setuptools`, so you can use `pyproject = true`
176 even if the package only has a `setup.py`. When set to `false`, you can
177 use the existing [hooks](#setup-hooks) or provide your own logic to build the
178 package. This can be useful for packages that don't support the pyproject
179 format. When unset, the legacy `setuptools` hooks are used for backwards
180 compatibility.
181* `makeWrapperArgs ? []`: A list of strings. Arguments to be passed to
182 [`makeWrapper`](#fun-makeWrapper), which wraps generated binaries. By default, the arguments to
183 [`makeWrapper`](#fun-makeWrapper) set `PATH` and `PYTHONPATH` environment variables before calling
184 the binary. Additional arguments here can allow a developer to set environment
185 variables which will be available when the binary is run. For example,
186 `makeWrapperArgs = ["--set FOO BAR" "--set BAZ QUX"]`.
187* `namePrefix`: Prepends text to `${name}` parameter. In case of libraries, this
188 defaults to `"python3.8-"` for Python 3.8, etc., and in case of applications to `""`.
189* `pipInstallFlags ? []`: A list of strings. Arguments to be passed to `pip
190 install`. To pass options to `python setup.py install`, use
191 `--install-option`. E.g., `pipInstallFlags=["--install-option='--cpp_implementation'"]`.
192* `pipBuildFlags ? []`: A list of strings. Arguments to be passed to `pip wheel`.
193* `pypaBuildFlags ? []`: A list of strings. Arguments to be passed to `python -m build --wheel`.
194* `pythonPath ? []`: List of packages to be added into `$PYTHONPATH`. Packages
195 in `pythonPath` are not propagated (contrary to [`propagatedBuildInputs`](#var-stdenv-propagatedBuildInputs)).
196* `preShellHook`: Hook to execute commands before `shellHook`.
197* `postShellHook`: Hook to execute commands after `shellHook`.
198* `removeBinByteCode ? true`: Remove bytecode from `/bin`. Bytecode is only
199 created when the filenames end with `.py`.
200* `setupPyGlobalFlags ? []`: List of flags passed to `setup.py` command.
201* `setupPyBuildFlags ? []`: List of flags passed to `setup.py build_ext` command.
202
203The [`stdenv.mkDerivation`](#sec-using-stdenv) function accepts various parameters for describing
204build inputs (see "Specifying dependencies"). The following are of special
205interest for Python packages, either because these are primarily used, or
206because their behaviour is different:
207
208* `nativeBuildInputs ? []`: Build-time only dependencies. Typically executables.
209* `build-system ? []`: Build-time only Python dependencies. Items listed in `build-system.requires`/`setup_requires`.
210* `buildInputs ? []`: Build and/or run-time dependencies that need to be
211 compiled for the host machine. Typically non-Python libraries which are being
212 linked.
213* `nativeCheckInputs ? []`: Dependencies needed for running the [`checkPhase`](#ssec-check-phase). These
214 are added to [`nativeBuildInputs`](#var-stdenv-nativeBuildInputs) when [`doCheck = true`](#var-stdenv-doCheck). Items listed in
215 `tests_require` go here.
216* `dependencies ? []`: Aside from propagating dependencies,
217 `buildPythonPackage` also injects code into and wraps executables with the
218 paths included in this list. Items listed in `install_requires` go here.
219* `optional-dependencies ? { }`: Optional feature flagged dependencies. Items listed in `extras_requires` go here.
220
221Aside from propagating dependencies,
222 `buildPythonPackage` also injects code into and wraps executables with the
223 paths included in this list. Items listed in `extras_requires` go here.
224
225##### Overriding Python packages {#overriding-python-packages}
226
227The `buildPythonPackage` function has a `overridePythonAttrs` method that can be
228used to override the package. In the following example we create an environment
229where we have the `blaze` package using an older version of `pandas`. We
230override first the Python interpreter and pass `packageOverrides` which contains
231the overrides for packages in the package set.
232
233```nix
234with import <nixpkgs> {};
235
236(let
237 python = let
238 packageOverrides = self: super: {
239 pandas = super.pandas.overridePythonAttrs(old: rec {
240 version = "0.19.1";
241 src = fetchPypi {
242 pname = "pandas";
243 inherit version;
244 hash = "sha256-JQn+rtpy/OA2deLszSKEuxyttqBzcAil50H+JDHUdCE=";
245 };
246 });
247 };
248 in pkgs.python3.override {inherit packageOverrides; self = python;};
249
250in python.withPackages(ps: [ ps.blaze ])).env
251```
252
253The next example shows a non trivial overriding of the `blas` implementation to
254be used through out all of the Python package set:
255
256```nix
257{
258 python3MyBlas = pkgs.python3.override {
259 packageOverrides = self: super: {
260 # We need toPythonModule for the package set to evaluate this
261 blas = super.toPythonModule(super.pkgs.blas.override {
262 blasProvider = super.pkgs.mkl;
263 });
264 lapack = super.toPythonModule(super.pkgs.lapack.override {
265 lapackProvider = super.pkgs.mkl;
266 });
267 };
268 };
269}
270```
271
272This is particularly useful for numpy and scipy users who want to gain speed with other blas implementations.
273Note that using `scipy = super.scipy.override { blas = super.pkgs.mkl; };` will likely result in
274compilation issues, because scipy dependencies need to use the same blas implementation as well.
275
276#### `buildPythonApplication` function {#buildpythonapplication-function}
277
278The [`buildPythonApplication`](#buildpythonapplication-function) function is practically the same as
279[`buildPythonPackage`](#buildpythonpackage-function). The main purpose of this function is to build a Python
280package where one is interested only in the executables, and not importable
281modules. For that reason, when adding this package to a [`python.buildEnv`](#python.buildenv-function), the
282modules won't be made available.
283
284Another difference is that [`buildPythonPackage`](#buildpythonpackage-function) by default prefixes the names of
285the packages with the version of the interpreter. Because this is irrelevant for
286applications, the prefix is omitted.
287
288When packaging a Python application with [`buildPythonApplication`](#buildpythonapplication-function), it should be
289called with `callPackage` and passed `python3` or `python3Packages` (possibly
290specifying an interpreter version), like this:
291
292```nix
293{ lib
294, python3Packages
295, fetchPypi
296}:
297
298python3Packages.buildPythonApplication rec {
299 pname = "luigi";
300 version = "2.7.9";
301 pyproject = true;
302
303 src = fetchPypi {
304 inherit pname version;
305 hash = "sha256-Pe229rT0aHwA98s+nTHQMEFKZPo/yw6sot8MivFDvAw=";
306 };
307
308 build-system = with python3Packages; [
309 setuptools
310 wheel
311 ];
312
313 dependencies = with python3Packages; [
314 tornado
315 python-daemon
316 ];
317
318 meta = {
319 # ...
320 };
321}
322```
323
324This is then added to `all-packages.nix` just as any other application would be.
325
326```nix
327{
328 luigi = callPackage ../applications/networking/cluster/luigi { };
329}
330```
331
332Since the package is an application, a consumer doesn't need to care about
333Python versions or modules, which is why they don't go in `python3Packages`.
334
335#### `toPythonApplication` function {#topythonapplication-function}
336
337A distinction is made between applications and libraries, however, sometimes a
338package is used as both. In this case the package is added as a library to
339`python-packages.nix` and as an application to `all-packages.nix`. To reduce
340duplication the `toPythonApplication` can be used to convert a library to an
341application.
342
343The Nix expression shall use [`buildPythonPackage`](#buildpythonpackage-function) and be called from
344`python-packages.nix`. A reference shall be created from `all-packages.nix` to
345the attribute in `python-packages.nix`, and the `toPythonApplication` shall be
346applied to the reference:
347
348```nix
349{
350 youtube-dl = with python3Packages; toPythonApplication youtube-dl;
351}
352```
353
354#### `toPythonModule` function {#topythonmodule-function}
355
356In some cases, such as bindings, a package is created using
357[`stdenv.mkDerivation`](#sec-using-stdenv) and added as attribute in `all-packages.nix`. The Python
358bindings should be made available from `python-packages.nix`. The
359`toPythonModule` function takes a derivation and makes certain Python-specific
360modifications.
361
362```nix
363{
364 opencv = toPythonModule (pkgs.opencv.override {
365 enablePython = true;
366 pythonPackages = self;
367 });
368}
369```
370
371Do pay attention to passing in the right Python version!
372
373#### `python.buildEnv` function {#python.buildenv-function}
374
375Python environments can be created using the low-level `pkgs.buildEnv` function.
376This example shows how to create an environment that has the Pyramid Web Framework.
377Saving the following as `default.nix`
378
379```nix
380with import <nixpkgs> {};
381
382python3.buildEnv.override {
383 extraLibs = [ python3Packages.pyramid ];
384 ignoreCollisions = true;
385}
386```
387
388and running `nix-build` will create
389
390```
391/nix/store/cf1xhjwzmdki7fasgr4kz6di72ykicl5-python-2.7.8-env
392```
393
394with wrapped binaries in `bin/`.
395
396You can also use the `env` attribute to create local environments with needed
397packages installed. This is somewhat comparable to `virtualenv`. For example,
398running `nix-shell` with the following `shell.nix`
399
400```nix
401with import <nixpkgs> {};
402
403(python3.buildEnv.override {
404 extraLibs = with python3Packages; [
405 numpy
406 requests
407 ];
408}).env
409```
410
411will drop you into a shell where Python will have the
412specified packages in its path.
413
414##### `python.buildEnv` arguments {#python.buildenv-arguments}
415
416
417* `extraLibs`: List of packages installed inside the environment.
418* `postBuild`: Shell command executed after the build of environment.
419* `ignoreCollisions`: Ignore file collisions inside the environment (default is `false`).
420* `permitUserSite`: Skip setting the `PYTHONNOUSERSITE` environment variable in
421 wrapped binaries in the environment.
422
423#### `python.withPackages` function {#python.withpackages-function}
424
425The [`python.withPackages`](#python.withpackages-function) function provides a simpler interface to the [`python.buildEnv`](#python.buildenv-function) functionality.
426It takes a function as an argument that is passed the set of python packages and returns the list
427of the packages to be included in the environment. Using the [`withPackages`](#python.withpackages-function) function, the previous
428example for the Pyramid Web Framework environment can be written like this:
429
430```nix
431with import <nixpkgs> {};
432
433python.withPackages (ps: [ ps.pyramid ])
434```
435
436[`withPackages`](#python.withpackages-function) passes the correct package set for the specific interpreter
437version as an argument to the function. In the above example, `ps` equals
438`pythonPackages`. But you can also easily switch to using python3:
439
440```nix
441with import <nixpkgs> {};
442
443python3.withPackages (ps: [ ps.pyramid ])
444```
445
446Now, `ps` is set to `python3Packages`, matching the version of the interpreter.
447
448As [`python.withPackages`](#python.withpackages-function) uses [`python.buildEnv`](#python.buildenv-function) under the hood, it also
449supports the `env` attribute. The `shell.nix` file from the previous section can
450thus be also written like this:
451
452```nix
453with import <nixpkgs> {};
454
455(python3.withPackages (ps: with ps; [
456 numpy
457 requests
458])).env
459```
460
461In contrast to [`python.buildEnv`](#python.buildenv-function), [`python.withPackages`](#python.withpackages-function) does not support the
462more advanced options such as `ignoreCollisions = true` or `postBuild`. If you
463need them, you have to use [`python.buildEnv`](#python.buildenv-function).
464
465Python 2 namespace packages may provide `__init__.py` that collide. In that case
466[`python.buildEnv`](#python.buildenv-function) should be used with `ignoreCollisions = true`.
467
468#### Setup hooks {#setup-hooks}
469
470The following are setup hooks specifically for Python packages. Most of these
471are used in [`buildPythonPackage`](#buildpythonpackage-function).
472
473- `eggUnpackhook` to move an egg to the correct folder so it can be installed
474 with the `eggInstallHook`
475- `eggBuildHook` to skip building for eggs.
476- `eggInstallHook` to install eggs.
477- `pipBuildHook` to build a wheel using `pip` and PEP 517. Note a build system
478 (e.g. `setuptools` or `flit`) should still be added as `build-system`.
479- `pypaBuildHook` to build a wheel using
480 [`pypa/build`](https://pypa-build.readthedocs.io/en/latest/index.html) and
481 PEP 517/518. Note a build system (e.g. `setuptools` or `flit`) should still
482 be added as `build-system`.
483- `pipInstallHook` to install wheels.
484- `pytestCheckHook` to run tests with `pytest`. See [example usage](#using-pytestcheckhook).
485- `pythonCatchConflictsHook` to fail if the package depends on two different versions of the same dependency.
486- `pythonImportsCheckHook` to check whether importing the listed modules works.
487- `pythonRelaxDepsHook` will relax Python dependencies restrictions for the package.
488 See [example usage](#using-pythonrelaxdepshook).
489- `pythonRemoveBinBytecode` to remove bytecode from the `/bin` folder.
490- `setuptoolsBuildHook` to build a wheel using `setuptools`.
491- `setuptoolsCheckHook` to run tests with `python setup.py test`.
492- `sphinxHook` to build documentation and manpages using Sphinx.
493- `venvShellHook` to source a Python 3 `venv` at the `venvDir` location. A
494 `venv` is created if it does not yet exist. `postVenvCreation` can be used to
495 to run commands only after venv is first created.
496- `wheelUnpackHook` to move a wheel to the correct folder so it can be installed
497 with the `pipInstallHook`.
498- `unittestCheckHook` will run tests with `python -m unittest discover`. See [example usage](#using-unittestcheckhook).
499
500## User Guide {#user-guide}
501
502### Using Python {#using-python}
503
504#### Overview {#overview}
505
506Several versions of the Python interpreter are available on Nix, as well as a
507high amount of packages. The attribute `python3` refers to the default
508interpreter, which is currently CPython 3.11. The attribute `python` refers to
509CPython 2.7 for backwards-compatibility. It is also possible to refer to
510specific versions, e.g. `python311` refers to CPython 3.11, and `pypy` refers to
511the default PyPy interpreter.
512
513Python is used a lot, and in different ways. This affects also how it is
514packaged. In the case of Python on Nix, an important distinction is made between
515whether the package is considered primarily an application, or whether it should
516be used as a library, i.e., of primary interest are the modules in
517`site-packages` that should be importable.
518
519In the Nixpkgs tree Python applications can be found throughout, depending on
520what they do, and are called from the main package set. Python libraries,
521however, are in separate sets, with one set per interpreter version.
522
523The interpreters have several common attributes. One of these attributes is
524`pkgs`, which is a package set of Python libraries for this specific
525interpreter. E.g., the `toolz` package corresponding to the default interpreter
526is `python3.pkgs.toolz`, and the CPython 3.11 version is `python311.pkgs.toolz`.
527The main package set contains aliases to these package sets, e.g.
528`pythonPackages` refers to `python.pkgs` and `python311Packages` to
529`python311.pkgs`.
530
531#### Installing Python and packages {#installing-python-and-packages}
532
533The Nix and NixOS manuals explain how packages are generally installed. In the
534case of Python and Nix, it is important to make a distinction between whether the
535package is considered an application or a library.
536
537Applications on Nix are typically installed into your user profile imperatively
538using `nix-env -i`, and on NixOS declaratively by adding the package name to
539`environment.systemPackages` in `/etc/nixos/configuration.nix`. Dependencies
540such as libraries are automatically installed and should not be installed
541explicitly.
542
543The same goes for Python applications. Python applications can be installed in
544your profile, and will be wrapped to find their exact library dependencies,
545without impacting other applications or polluting your user environment.
546
547But Python libraries you would like to use for development cannot be installed,
548at least not individually, because they won't be able to find each other
549resulting in import errors. Instead, it is possible to create an environment
550with [`python.buildEnv`](#python.buildenv-function) or [`python.withPackages`](#python.withpackages-function) where the interpreter and other
551executables are wrapped to be able to find each other and all of the modules.
552
553In the following examples we will start by creating a simple, ad-hoc environment
554with a nix-shell that has `numpy` and `toolz` in Python 3.11; then we will create
555a re-usable environment in a single-file Python script; then we will create a
556full Python environment for development with this same environment.
557
558Philosophically, this should be familiar to users who are used to a `venv` style
559of development: individual projects create their own Python environments without
560impacting the global environment or each other.
561
562#### Ad-hoc temporary Python environment with `nix-shell` {#ad-hoc-temporary-python-environment-with-nix-shell}
563
564The simplest way to start playing with the way nix wraps and sets up Python
565environments is with `nix-shell` at the cmdline. These environments create a
566temporary shell session with a Python and a *precise* list of packages (plus
567their runtime dependencies), with no other Python packages in the Python
568interpreter's scope.
569
570To create a Python 3.11 session with `numpy` and `toolz` available, run:
571
572```sh
573$ nix-shell -p 'python311.withPackages(ps: with ps; [ numpy toolz ])'
574```
575
576By default `nix-shell` will start a `bash` session with this interpreter in our
577`PATH`, so if we then run:
578
579```Python console
580[nix-shell:~/src/nixpkgs]$ python3
581Python 3.11.3 (main, Apr 4 2023, 22:36:41) [GCC 12.2.0] on linux
582Type "help", "copyright", "credits" or "license" for more information.
583>>> import numpy; import toolz
584```
585
586Note that no other modules are in scope, even if they were imperatively
587installed into our user environment as a dependency of a Python application:
588
589```Python console
590>>> import requests
591Traceback (most recent call last):
592 File "<stdin>", line 1, in <module>
593ModuleNotFoundError: No module named 'requests'
594```
595
596We can add as many additional modules onto the `nix-shell` as we need, and we
597will still get 1 wrapped Python interpreter. We can start the interpreter
598directly like so:
599
600```sh
601$ nix-shell -p "python311.withPackages (ps: with ps; [ numpy toolz requests ])" --run python3
602this derivation will be built:
603 /nix/store/r19yf5qgfiakqlhkgjahbg3zg79549n4-python3-3.11.2-env.drv
604building '/nix/store/r19yf5qgfiakqlhkgjahbg3zg79549n4-python3-3.11.2-env.drv'...
605created 273 symlinks in user environment
606Python 3.11.2 (main, Feb 7 2023, 13:52:42) [GCC 12.2.0] on linux
607Type "help", "copyright", "credits" or "license" for more information.
608>>> import requests
609>>>
610```
611
612Notice that this time it built a new Python environment, which now includes
613`requests`. Building an environment just creates wrapper scripts that expose the
614selected dependencies to the interpreter while re-using the actual modules. This
615means if any other env has installed `requests` or `numpy` in a different
616context, we don't need to recompile them -- we just recompile the wrapper script
617that sets up an interpreter pointing to them. This matters much more for "big"
618modules like `pytorch` or `tensorflow`.
619
620Module names usually match their names on [pypi.org](https://pypi.org/), but
621you can use the [Nixpkgs search website](https://nixos.org/nixos/packages.html)
622to find them as well (along with non-python packages).
623
624At this point we can create throwaway experimental Python environments with
625arbitrary dependencies. This is a good way to get a feel for how the Python
626interpreter and dependencies work in Nix and NixOS, but to do some actual
627development, we'll want to make it a bit more persistent.
628
629##### Running Python scripts and using `nix-shell` as shebang {#running-python-scripts-and-using-nix-shell-as-shebang}
630
631Sometimes, we have a script whose header looks like this:
632
633```python
634#!/usr/bin/env python3
635import numpy as np
636a = np.array([1,2])
637b = np.array([3,4])
638print(f"The dot product of {a} and {b} is: {np.dot(a, b)}")
639```
640
641Executing this script requires a `python3` that has `numpy`. Using what we learned
642in the previous section, we could startup a shell and just run it like so:
643
644```ShellSession
645$ nix-shell -p 'python311.withPackages (ps: with ps; [ numpy ])' --run 'python3 foo.py'
646The dot product of [1 2] and [3 4] is: 11
647```
648
649But if we maintain the script ourselves, and if there are more dependencies, it
650may be nice to encode those dependencies in source to make the script re-usable
651without that bit of knowledge. That can be done by using `nix-shell` as a
652[shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)), like so:
653
654```python
655#!/usr/bin/env nix-shell
656#!nix-shell -i python3 -p "python3.withPackages(ps: [ ps.numpy ])"
657import numpy as np
658a = np.array([1,2])
659b = np.array([3,4])
660print(f"The dot product of {a} and {b} is: {np.dot(a, b)}")
661```
662
663Then we execute it, without requiring any environment setup at all!
664
665```sh
666$ ./foo.py
667The dot product of [1 2] and [3 4] is: 11
668```
669
670If the dependencies are not available on the host where `foo.py` is executed, it
671will build or download them from a Nix binary cache prior to starting up, prior
672that it is executed on a machine with a multi-user nix installation.
673
674This provides a way to ship a self bootstrapping Python script, akin to a
675statically linked binary, where it can be run on any machine (provided nix is
676installed) without having to assume that `numpy` is installed globally on the
677system.
678
679By default it is pulling the import checkout of Nixpkgs itself from our nix
680channel, which is nice as it cache aligns with our other package builds, but we
681can make it fully reproducible by pinning the `nixpkgs` import:
682
683```python
684#!/usr/bin/env nix-shell
685#!nix-shell -i python3 -p "python3.withPackages (ps: [ ps.numpy ])"
686#!nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/e51209796c4262bfb8908e3d6d72302fe4e96f5f.tar.gz
687import numpy as np
688a = np.array([1,2])
689b = np.array([3,4])
690print(f"The dot product of {a} and {b} is: {np.dot(a, b)}")
691```
692
693This will execute with the exact same versions of Python 3.10, numpy, and system
694dependencies a year from now as it does today, because it will always use
695exactly git commit `e51209796c4262bfb8908e3d6d72302fe4e96f5f` of Nixpkgs for all
696of the package versions.
697
698This is also a great way to ensure the script executes identically on different
699servers.
700
701##### Load environment from `.nix` expression {#load-environment-from-.nix-expression}
702
703We've now seen how to create an ad-hoc temporary shell session, and how to
704create a single script with Python dependencies, but in the course of normal
705development we're usually working in an entire package repository.
706
707As explained [in the `nix-shell` section](https://nixos.org/manual/nix/stable/command-ref/nix-shell) of the Nix manual, `nix-shell` can also load an expression from a `.nix` file.
708Say we want to have Python 3.11, `numpy` and `toolz`, like before,
709in an environment. We can add a `shell.nix` file describing our dependencies:
710
711```nix
712with import <nixpkgs> {};
713(python311.withPackages (ps: with ps; [
714 numpy
715 toolz
716])).env
717```
718
719And then at the command line, just typing `nix-shell` produces the same
720environment as before. In a normal project, we'll likely have many more
721dependencies; this can provide a way for developers to share the environments
722with each other and with CI builders.
723
724What's happening here?
725
7261. We begin with importing the Nix Packages collections. `import <nixpkgs>`
727 imports the `<nixpkgs>` function, `{}` calls it and the `with` statement
728 brings all attributes of `nixpkgs` in the local scope. These attributes form
729 the main package set.
7302. Then we create a Python 3.11 environment with the [`withPackages`](#python.withpackages-function) function, as before.
7313. The [`withPackages`](#python.withpackages-function) function expects us to provide a function as an argument
732 that takes the set of all Python packages and returns a list of packages to
733 include in the environment. Here, we select the packages `numpy` and `toolz`
734 from the package set.
735
736To combine this with `mkShell` you can:
737
738```nix
739with import <nixpkgs> {};
740let
741 pythonEnv = python311.withPackages (ps: [
742 ps.numpy
743 ps.toolz
744 ]);
745in mkShell {
746 packages = [
747 pythonEnv
748
749 black
750 mypy
751
752 libffi
753 openssl
754 ];
755}
756```
757
758This will create a unified environment that has not just our Python interpreter
759and its Python dependencies, but also tools like `black` or `mypy` and libraries
760like `libffi` the `openssl` in scope. This is generic and can span any number of
761tools or languages across the Nixpkgs ecosystem.
762
763##### Installing environments globally on the system {#installing-environments-globally-on-the-system}
764
765Up to now, we've been creating environments scoped to an ad-hoc shell session,
766or a single script, or a single project. This is generally advisable, as it
767avoids pollution across contexts.
768
769However, sometimes we know we will often want a Python with some basic packages,
770and want this available without having to enter into a shell or build context.
771This can be useful to have things like vim/emacs editors and plugins or shell
772tools "just work" without having to set them up, or when running other software
773that expects packages to be installed globally.
774
775To create your own custom environment, create a file in `~/.config/nixpkgs/overlays/`
776that looks like this:
777
778```nix
779# ~/.config/nixpkgs/overlays/myEnv.nix
780self: super: {
781 myEnv = super.buildEnv {
782 name = "myEnv";
783 paths = [
784 # A Python 3 interpreter with some packages
785 (self.python3.withPackages (
786 ps: with ps; [
787 pyflakes
788 pytest
789 black
790 ]
791 ))
792
793 # Some other packages we'd like as part of this env
794 self.mypy
795 self.black
796 self.ripgrep
797 self.tmux
798 ];
799 };
800}
801```
802
803You can then build and install this to your profile with:
804
805```sh
806nix-env -iA myEnv
807```
808
809One limitation of this is that you can only have 1 Python env installed
810globally, since they conflict on the `python` to load out of your `PATH`.
811
812If you get a conflict or prefer to keep the setup clean, you can have `nix-env`
813atomically *uninstall* all other imperatively installed packages and replace
814your profile with just `myEnv` by using the `--replace` flag.
815
816##### Environment defined in `/etc/nixos/configuration.nix` {#environment-defined-in-etcnixosconfiguration.nix}
817
818For the sake of completeness, here's how to install the environment system-wide
819on NixOS.
820
821```nix
822{ # ...
823
824 environment.systemPackages = with pkgs; [
825 (python310.withPackages(ps: with ps; [ numpy toolz ]))
826 ];
827}
828```
829
830### Developing with Python {#developing-with-python}
831
832Above, we were mostly just focused on use cases and what to do to get started
833creating working Python environments in nix.
834
835Now that you know the basics to be up and running, it is time to take a step
836back and take a deeper look at how Python packages are packaged on Nix.
837
838#### Python library packages in Nixpkgs {#python-library-packages-in-nixpkgs}
839
840With Nix all packages are built by functions. The main function in Nix for
841building Python libraries is [`buildPythonPackage`](#buildpythonpackage-function). Let's see how we can build the
842`toolz` package.
843
844```nix
845{ lib
846, buildPythonPackage
847, fetchPypi
848, setuptools
849, wheel
850}:
851
852buildPythonPackage rec {
853 pname = "toolz";
854 version = "0.10.0";
855 pyproject = true;
856
857 src = fetchPypi {
858 inherit pname version;
859 hash = "sha256-CP3V73yWSArRHBLUct4hrNMjWZlvaaUlkpm1QP66RWA=";
860 };
861
862 build-system = [
863 setuptools
864 wheel
865 ];
866
867 # has no tests
868 doCheck = false;
869
870 pythonImportsCheck = [
871 "toolz.itertoolz"
872 "toolz.functoolz"
873 "toolz.dicttoolz"
874 ];
875
876 meta = {
877 changelog = "https://github.com/pytoolz/toolz/releases/tag/${version}";
878 homepage = "https://github.com/pytoolz/toolz";
879 description = "List processing tools and functional utilities";
880 license = lib.licenses.bsd3;
881 };
882}
883```
884
885What happens here? The function [`buildPythonPackage`](#buildpythonpackage-function) is called and as argument
886it accepts a set. In this case the set is a recursive set, `rec`. One of the
887arguments is the name of the package, which consists of a basename (generally
888following the name on PyPi) and a version. Another argument, `src` specifies the
889source, which in this case is fetched from PyPI using the helper function
890`fetchPypi`. The argument `doCheck` is used to set whether tests should be run
891when building the package. Since there are no tests, we rely on [`pythonImportsCheck`](#using-pythonimportscheck)
892to test whether the package can be imported. Furthermore, we specify some meta
893information. The output of the function is a derivation.
894
895An expression for `toolz` can be found in the Nixpkgs repository. As explained
896in the introduction of this Python section, a derivation of `toolz` is available
897for each interpreter version, e.g. `python311.pkgs.toolz` refers to the `toolz`
898derivation corresponding to the CPython 3.11 interpreter.
899
900The above example works when you're directly working on
901`pkgs/top-level/python-packages.nix` in the Nixpkgs repository. Often though,
902you will want to test a Nix expression outside of the Nixpkgs tree.
903
904The following expression creates a derivation for the `toolz` package,
905and adds it along with a `numpy` package to a Python environment.
906
907```nix
908with import <nixpkgs> {};
909
910( let
911 my_toolz = python311.pkgs.buildPythonPackage rec {
912 pname = "toolz";
913 version = "0.10.0";
914 pyproject = true;
915
916 src = fetchPypi {
917 inherit pname version;
918 hash = "sha256-CP3V73yWSArRHBLUct4hrNMjWZlvaaUlkpm1QP66RWA=";
919 };
920
921 build-system = [
922 python311.pkgs.setuptools
923 python311.pkgs.wheel
924 ];
925
926 # has no tests
927 doCheck = false;
928
929 meta = {
930 homepage = "https://github.com/pytoolz/toolz/";
931 description = "List processing tools and functional utilities";
932 # [...]
933 };
934 };
935
936 in python311.withPackages (ps: with ps; [
937 numpy
938 my_toolz
939 ])
940).env
941```
942
943Executing `nix-shell` will result in an environment in which you can use
944Python 3.11 and the `toolz` package. As you can see we had to explicitly mention
945for which Python version we want to build a package.
946
947So, what did we do here? Well, we took the Nix expression that we used earlier
948to build a Python environment, and said that we wanted to include our own
949version of `toolz`, named `my_toolz`. To introduce our own package in the scope
950of [`withPackages`](#python.withpackages-function) we used a `let` expression. You can see that we used
951`ps.numpy` to select numpy from the nixpkgs package set (`ps`). We did not take
952`toolz` from the Nixpkgs package set this time, but instead took our own version
953that we introduced with the `let` expression.
954
955#### Handling dependencies {#handling-dependencies}
956
957Our example, `toolz`, does not have any dependencies on other Python packages or system libraries.
958[`buildPythonPackage`](#buildpythonpackage-function) uses the the following arguments in the following circumstances:
959
960- `dependencies` - For Python runtime dependencies.
961- `build-system` - For Python build-time requirements.
962- [`buildInputs`](#var-stdenv-buildInputs) - For non-Python build-time requirements.
963- [`nativeCheckInputs`](#var-stdenv-nativeCheckInputs) - For test dependencies
964
965Dependencies can belong to multiple arguments, for example if something is both a build time requirement & a runtime dependency.
966
967The following example shows which arguments are given to [`buildPythonPackage`](#buildpythonpackage-function) in
968order to build [`datashape`](https://github.com/blaze/datashape).
969
970```nix
971{ lib
972, buildPythonPackage
973, fetchPypi
974
975# build dependencies
976, setuptools, wheel
977
978# dependencies
979, numpy, multipledispatch, python-dateutil
980
981# tests
982, pytest
983}:
984
985buildPythonPackage rec {
986 pname = "datashape";
987 version = "0.4.7";
988 pyproject = true;
989
990 src = fetchPypi {
991 inherit pname version;
992 hash = "sha256-FLLvdm1MllKrgTGC6Gb0k0deZeVYvtCCLji/B7uhong=";
993 };
994
995 build-system = [
996 setuptools
997 wheel
998 ];
999
1000 dependencies = [
1001 multipledispatch
1002 numpy
1003 python-dateutil
1004 ];
1005
1006 nativeCheckInputs = [
1007 pytest
1008 ];
1009
1010 meta = {
1011 changelog = "https://github.com/blaze/datashape/releases/tag/${version}";
1012 homepage = "https://github.com/ContinuumIO/datashape";
1013 description = "A data description language";
1014 license = lib.licenses.bsd2;
1015 };
1016}
1017```
1018
1019We can see several runtime dependencies, `numpy`, `multipledispatch`, and
1020`python-dateutil`. Furthermore, we have [`nativeCheckInputs`](#var-stdenv-nativeCheckInputs) with `pytest`.
1021`pytest` is a test runner and is only used during the [`checkPhase`](#ssec-check-phase) and is
1022therefore not added to `dependencies`.
1023
1024In the previous case we had only dependencies on other Python packages to consider.
1025Occasionally you have also system libraries to consider. E.g., `lxml` provides
1026Python bindings to `libxml2` and `libxslt`. These libraries are only required
1027when building the bindings and are therefore added as [`buildInputs`](#var-stdenv-buildInputs).
1028
1029```nix
1030{ lib
1031, buildPythonPackage
1032, fetchPypi
1033, setuptools
1034, wheel
1035, libxml2
1036, libxslt
1037}:
1038
1039buildPythonPackage rec {
1040 pname = "lxml";
1041 version = "3.4.4";
1042 pyproject = true;
1043
1044 src = fetchPypi {
1045 inherit pname version;
1046 hash = "sha256-s9NiusRxFydHzaNRMjjxFcvWxfi45jGb9ql6eJJyQJk=";
1047 };
1048
1049 build-system = [
1050 setuptools
1051 wheel
1052 ];
1053
1054 buildInputs = [
1055 libxml2
1056 libxslt
1057 ];
1058
1059 meta = {
1060 changelog = "https://github.com/lxml/lxml/releases/tag/lxml-${version}";
1061 description = "Pythonic binding for the libxml2 and libxslt libraries";
1062 homepage = "https://lxml.de";
1063 license = lib.licenses.bsd3;
1064 maintainers = with lib.maintainers; [ sjourdois ];
1065 };
1066}
1067```
1068
1069In this example `lxml` and Nix are able to work out exactly where the relevant
1070files of the dependencies are. This is not always the case.
1071
1072The example below shows bindings to The Fastest Fourier Transform in the West,
1073commonly known as FFTW. On Nix we have separate packages of FFTW for the
1074different types of floats (`"single"`, `"double"`, `"long-double"`). The
1075bindings need all three types, and therefore we add all three as [`buildInputs`](#var-stdenv-buildInputs).
1076The bindings don't expect to find each of them in a different folder, and
1077therefore we have to set `LDFLAGS` and `CFLAGS`.
1078
1079```nix
1080{ lib
1081, buildPythonPackage
1082, fetchPypi
1083
1084# build dependencies
1085, setuptools
1086, wheel
1087
1088# dependencies
1089, fftw
1090, fftwFloat
1091, fftwLongDouble
1092, numpy
1093, scipy
1094}:
1095
1096buildPythonPackage rec {
1097 pname = "pyFFTW";
1098 version = "0.9.2";
1099 pyproject = true;
1100
1101 src = fetchPypi {
1102 inherit pname version;
1103 hash = "sha256-9ru2r6kwhUCaskiFoaPNuJCfCVoUL01J40byvRt4kHQ=";
1104 };
1105
1106 build-system = [
1107 setuptools
1108 wheel
1109 ];
1110
1111 buildInputs = [
1112 fftw
1113 fftwFloat
1114 fftwLongDouble
1115 ];
1116
1117 dependencies = [
1118 numpy
1119 scipy
1120 ];
1121
1122 preConfigure = ''
1123 export LDFLAGS="-L${fftw.dev}/lib -L${fftwFloat.out}/lib -L${fftwLongDouble.out}/lib"
1124 export CFLAGS="-I${fftw.dev}/include -I${fftwFloat.dev}/include -I${fftwLongDouble.dev}/include"
1125 '';
1126
1127 # Tests cannot import pyfftw. pyfftw works fine though.
1128 doCheck = false;
1129
1130 meta = {
1131 changelog = "https://github.com/pyFFTW/pyFFTW/releases/tag/v${version}";
1132 description = "A pythonic wrapper around FFTW, the FFT library, presenting a unified interface for all the supported transforms";
1133 homepage = "http://hgomersall.github.com/pyFFTW";
1134 license = with lib.licenses; [ bsd2 bsd3 ];
1135 };
1136}
1137```
1138
1139Note also the line [`doCheck = false;`](#var-stdenv-doCheck), we explicitly disabled running the test-suite.
1140
1141#### Testing Python Packages {#testing-python-packages}
1142
1143It is highly encouraged to have testing as part of the package build. This
1144helps to avoid situations where the package was able to build and install,
1145but is not usable at runtime. Currently, all packages will use the `test`
1146command provided by the setup.py (i.e. `python setup.py test`). However,
1147this is currently deprecated https://github.com/pypa/setuptools/pull/1878
1148and your package should provide its own [`checkPhase`](#ssec-check-phase).
1149
1150::: {.note}
1151The [`checkPhase`](#ssec-check-phase) for python maps to the `installCheckPhase` on a
1152normal derivation. This is due to many python packages not behaving well
1153to the pre-installed version of the package. Version info, and natively
1154compiled extensions generally only exist in the install directory, and
1155thus can cause issues when a test suite asserts on that behavior.
1156:::
1157
1158::: {.note}
1159Tests should only be disabled if they don't agree with nix
1160(e.g. external dependencies, network access, flakey tests), however,
1161as many tests should be enabled as possible. Failing tests can still be
1162a good indication that the package is not in a valid state.
1163:::
1164
1165#### Using pytest {#using-pytest}
1166
1167Pytest is the most common test runner for python repositories. A trivial
1168test run would be:
1169
1170```nix
1171{
1172 nativeCheckInputs = [ pytest ];
1173 checkPhase = ''
1174 runHook preCheck
1175
1176 pytest
1177
1178 runHook postCheck
1179 '';
1180}
1181```
1182
1183However, many repositories' test suites do not translate well to nix's build
1184sandbox, and will generally need many tests to be disabled.
1185
1186To filter tests using pytest, one can do the following:
1187
1188```nix
1189{
1190 nativeCheckInputs = [ pytest ];
1191 # avoid tests which need additional data or touch network
1192 checkPhase = ''
1193 runHook preCheck
1194
1195 pytest tests/ --ignore=tests/integration -k 'not download and not update' --ignore=tests/test_failing.py
1196
1197 runHook postCheck
1198 '';
1199}
1200```
1201
1202`--ignore` will tell pytest to ignore that file or directory from being
1203collected as part of a test run. This is useful is a file uses a package
1204which is not available in nixpkgs, thus skipping that test file is much
1205easier than having to create a new package.
1206
1207`-k` is used to define a predicate for test names. In this example, we are
1208filtering out tests which contain `download` or `update` in their test case name.
1209Only one `-k` argument is allowed, and thus a long predicate should be concatenated
1210with “\\” and wrapped to the next line.
1211
1212::: {.note}
1213In pytest==6.0.1, the use of “\\” to continue a line (e.g. `-k 'not download \'`) has
1214been removed, in this case, it's recommended to use `pytestCheckHook`.
1215:::
1216
1217#### Using pytestCheckHook {#using-pytestcheckhook}
1218
1219`pytestCheckHook` is a convenient hook which will substitute the setuptools
1220`test` command for a [`checkPhase`](#ssec-check-phase) which runs `pytest`. This is also beneficial
1221when a package may need many items disabled to run the test suite.
1222
1223Using the example above, the analogous `pytestCheckHook` usage would be:
1224
1225```nix
1226{
1227 nativeCheckInputs = [
1228 pytestCheckHook
1229 ];
1230
1231 # requires additional data
1232 pytestFlagsArray = [
1233 "tests/"
1234 "--ignore=tests/integration"
1235 ];
1236
1237 disabledTests = [
1238 # touches network
1239 "download"
1240 "update"
1241 ];
1242
1243 disabledTestPaths = [
1244 "tests/test_failing.py"
1245 ];
1246}
1247```
1248
1249This is especially useful when tests need to be conditionally disabled,
1250for example:
1251
1252```nix
1253{
1254 disabledTests = [
1255 # touches network
1256 "download"
1257 "update"
1258 ] ++ lib.optionals (pythonAtLeast "3.8") [
1259 # broken due to python3.8 async changes
1260 "async"
1261 ] ++ lib.optionals stdenv.isDarwin [
1262 # can fail when building with other packages
1263 "socket"
1264 ];
1265}
1266```
1267
1268Trying to concatenate the related strings to disable tests in a regular
1269[`checkPhase`](#ssec-check-phase) would be much harder to read. This also enables us to comment on
1270why specific tests are disabled.
1271
1272#### Using pythonImportsCheck {#using-pythonimportscheck}
1273
1274Although unit tests are highly preferred to validate correctness of a package, not
1275all packages have test suites that can be run easily, and some have none at all.
1276To help ensure the package still works, [`pythonImportsCheck`](#using-pythonimportscheck) can attempt to import
1277the listed modules.
1278
1279```nix
1280{
1281 pythonImportsCheck = [
1282 "requests"
1283 "urllib"
1284 ];
1285}
1286```
1287
1288roughly translates to:
1289
1290```nix
1291{
1292 postCheck = ''
1293 PYTHONPATH=$out/${python.sitePackages}:$PYTHONPATH
1294 python -c "import requests; import urllib"
1295 '';
1296}
1297```
1298
1299However, this is done in its own phase, and not dependent on whether [`doCheck = true;`](#var-stdenv-doCheck).
1300
1301This can also be useful in verifying that the package doesn't assume commonly
1302present packages (e.g. `setuptools`).
1303
1304#### Using pythonRelaxDepsHook {#using-pythonrelaxdepshook}
1305
1306It is common for upstream to specify a range of versions for its package
1307dependencies. This makes sense, since it ensures that the package will be built
1308with a subset of packages that is well tested. However, this commonly causes
1309issues when packaging in Nixpkgs, because the dependencies that this package
1310may need are too new or old for the package to build correctly. We also cannot
1311package multiple versions of the same package since this may cause conflicts
1312in `PYTHONPATH`.
1313
1314One way to side step this issue is to relax the dependencies. This can be done
1315by either removing the package version range or by removing the package
1316declaration entirely. This can be done using the `pythonRelaxDepsHook` hook. For
1317example, given the following `requirements.txt` file:
1318
1319```
1320pkg1<1.0
1321pkg2
1322pkg3>=1.0,<=2.0
1323```
1324
1325we can do:
1326
1327```nix
1328{
1329 nativeBuildInputs = [
1330 pythonRelaxDepsHook
1331 ];
1332 pythonRelaxDeps = [
1333 "pkg1"
1334 "pkg3"
1335 ];
1336 pythonRemoveDeps = [
1337 "pkg2"
1338 ];
1339}
1340```
1341
1342which would result in the following `requirements.txt` file:
1343
1344```
1345pkg1
1346pkg3
1347```
1348
1349Another option is to pass `true`, that will relax/remove all dependencies, for
1350example:
1351
1352```nix
1353{
1354 nativeBuildInputs = [ pythonRelaxDepsHook ];
1355 pythonRelaxDeps = true;
1356}
1357```
1358
1359which would result in the following `requirements.txt` file:
1360
1361```
1362pkg1
1363pkg2
1364pkg3
1365```
1366
1367In general you should always use `pythonRelaxDeps`, because `pythonRemoveDeps`
1368will convert build errors into runtime errors. However `pythonRemoveDeps` may
1369still be useful in exceptional cases, and also to remove dependencies wrongly
1370declared by upstream (for example, declaring `black` as a runtime dependency
1371instead of a dev dependency).
1372
1373Keep in mind that while the examples above are done with `requirements.txt`,
1374`pythonRelaxDepsHook` works by modifying the resulting wheel file, so it should
1375work with any of the [existing hooks](#setup-hooks).
1376
1377#### Using unittestCheckHook {#using-unittestcheckhook}
1378
1379`unittestCheckHook` is a hook which will substitute the setuptools `test` command for a [`checkPhase`](#ssec-check-phase) which runs `python -m unittest discover`:
1380
1381```nix
1382{
1383 nativeCheckInputs = [
1384 unittestCheckHook
1385 ];
1386
1387 unittestFlagsArray = [
1388 "-s" "tests" "-v"
1389 ];
1390}
1391```
1392
1393#### Using sphinxHook {#using-sphinxhook}
1394
1395The `sphinxHook` is a helpful tool to build documentation and manpages
1396using the popular Sphinx documentation generator.
1397It is setup to automatically find common documentation source paths and
1398render them using the default `html` style.
1399
1400```nix
1401{
1402 outputs = [
1403 "out"
1404 "doc"
1405 ];
1406
1407 nativeBuildInputs = [
1408 sphinxHook
1409 ];
1410}
1411```
1412
1413The hook will automatically build and install the artifact into the
1414`doc` output, if it exists. It also provides an automatic diversion
1415for the artifacts of the `man` builder into the `man` target.
1416
1417```nix
1418{
1419 outputs = [
1420 "out"
1421 "doc"
1422 "man"
1423 ];
1424
1425 # Use multiple builders
1426 sphinxBuilders = [
1427 "singlehtml"
1428 "man"
1429 ];
1430}
1431```
1432
1433Overwrite `sphinxRoot` when the hook is unable to find your
1434documentation source root.
1435
1436```nix
1437{
1438 # Configure sphinxRoot for uncommon paths
1439 sphinxRoot = "weird/docs/path";
1440}
1441```
1442
1443The hook is also available to packages outside the python ecosystem by
1444referencing it using `sphinxHook` from top-level.
1445
1446### Organising your packages {#organising-your-packages}
1447
1448So far we discussed how you can use Python on Nix, and how you can develop with
1449it. We've looked at how you write expressions to package Python packages, and we
1450looked at how you can create environments in which specified packages are
1451available.
1452
1453At some point you'll likely have multiple packages which you would
1454like to be able to use in different projects. In order to minimise unnecessary
1455duplication we now look at how you can maintain a repository with your
1456own packages. The important functions here are `import` and `callPackage`.
1457
1458### Including a derivation using `callPackage` {#including-a-derivation-using-callpackage}
1459
1460Earlier we created a Python environment using [`withPackages`](#python.withpackages-function), and included the
1461`toolz` package via a `let` expression.
1462Let's split the package definition from the environment definition.
1463
1464We first create a function that builds `toolz` in `~/path/to/toolz/release.nix`
1465
1466```nix
1467{ lib
1468, buildPythonPackage
1469, fetchPypi
1470, setuptools
1471, wheel
1472}:
1473
1474buildPythonPackage rec {
1475 pname = "toolz";
1476 version = "0.10.0";
1477 pyproject = true;
1478
1479 src = fetchPypi {
1480 inherit pname version;
1481 hash = "sha256-CP3V73yWSArRHBLUct4hrNMjWZlvaaUlkpm1QP66RWA=";
1482 };
1483
1484 build-system = [
1485 setuptools
1486 wheel
1487 ];
1488
1489 meta = {
1490 changelog = "https://github.com/pytoolz/toolz/releases/tag/${version}";
1491 homepage = "https://github.com/pytoolz/toolz/";
1492 description = "List processing tools and functional utilities";
1493 license = lib.licenses.bsd3;
1494 };
1495}
1496```
1497
1498It takes an argument [`buildPythonPackage`](#buildpythonpackage-function). We now call this function using
1499`callPackage` in the definition of our environment
1500
1501```nix
1502with import <nixpkgs> {};
1503
1504( let
1505 toolz = callPackage /path/to/toolz/release.nix {
1506 buildPythonPackage = python310
1507Packages.buildPythonPackage;
1508 };
1509 in python310.withPackages (ps: [
1510 ps.numpy
1511 toolz
1512 ])
1513).env
1514```
1515
1516Important to remember is that the Python version for which the package is made
1517depends on the `python` derivation that is passed to [`buildPythonPackage`](#buildpythonpackage-function). Nix
1518tries to automatically pass arguments when possible, which is why generally you
1519don't explicitly define which `python` derivation should be used. In the above
1520example we use [`buildPythonPackage`](#buildpythonpackage-function) that is part of the set `python3Packages`,
1521and in this case the `python3` interpreter is automatically used.
1522
1523## FAQ {#faq}
1524
1525### How to solve circular dependencies? {#how-to-solve-circular-dependencies}
1526
1527Consider the packages `A` and `B` that depend on each other. When packaging `B`,
1528a solution is to override package `A` not to depend on `B` as an input. The same
1529should also be done when packaging `A`.
1530
1531### How to override a Python package? {#how-to-override-a-python-package}
1532
1533We can override the interpreter and pass `packageOverrides`. In the following
1534example we rename the `pandas` package and build it.
1535
1536```nix
1537with import <nixpkgs> {};
1538
1539(let
1540 python = let
1541 packageOverrides = self: super: {
1542 pandas = super.pandas.overridePythonAttrs(old: {name="foo";});
1543 };
1544 in pkgs.python310.override {
1545 inherit packageOverrides;
1546 };
1547
1548in python.withPackages (ps: [
1549 ps.pandas
1550])).env
1551```
1552
1553Using `nix-build` on this expression will build an environment that contains the
1554package `pandas` but with the new name `foo`.
1555
1556All packages in the package set will use the renamed package. A typical use case
1557is to switch to another version of a certain package. For example, in the
1558Nixpkgs repository we have multiple versions of `django` and `scipy`. In the
1559following example we use a different version of `scipy` and create an
1560environment that uses it. All packages in the Python package set will now use
1561the updated `scipy` version.
1562
1563```nix
1564with import <nixpkgs> {};
1565
1566( let
1567 packageOverrides = self: super: {
1568 scipy = super.scipy_0_17;
1569 };
1570 in (pkgs.python310.override {
1571 inherit packageOverrides;
1572 }).withPackages (ps: [
1573 ps.blaze
1574 ])
1575).env
1576```
1577
1578The requested package `blaze` depends on `pandas` which itself depends on `scipy`.
1579
1580If you want the whole of Nixpkgs to use your modifications, then you can use
1581`overlays` as explained in this manual. In the following example we build a
1582`inkscape` using a different version of `numpy`.
1583
1584```nix
1585let
1586 pkgs = import <nixpkgs> {};
1587 newpkgs = import pkgs.path { overlays = [ (self: super: {
1588 python310 = let
1589 packageOverrides = python-self: python-super: {
1590 numpy = python-super.numpy_1_18;
1591 };
1592 in super.python310.override {inherit packageOverrides;};
1593 } ) ]; };
1594in newpkgs.inkscape
1595```
1596
1597### `python setup.py bdist_wheel` cannot create .whl {#python-setup.py-bdist_wheel-cannot-create-.whl}
1598
1599Executing `python setup.py bdist_wheel` in a `nix-shell`fails with
1600
1601```
1602ValueError: ZIP does not support timestamps before 1980
1603```
1604
1605This is because files from the Nix store (which have a timestamp of the UNIX
1606epoch of January 1, 1970) are included in the .ZIP, but .ZIP archives follow the
1607DOS convention of counting timestamps from 1980.
1608
1609The command `bdist_wheel` reads the `SOURCE_DATE_EPOCH` environment variable,
1610which `nix-shell` sets to 1. Unsetting this variable or giving it a value
1611corresponding to 1980 or later enables building wheels.
1612
1613Use 1980 as timestamp:
1614
1615```shell
1616nix-shell --run "SOURCE_DATE_EPOCH=315532800 python3 setup.py bdist_wheel"
1617```
1618
1619or the current time:
1620
1621```shell
1622nix-shell --run "SOURCE_DATE_EPOCH=$(date +%s) python3 setup.py bdist_wheel"
1623```
1624
1625or unset `SOURCE_DATE_EPOCH`:
1626
1627```shell
1628nix-shell --run "unset SOURCE_DATE_EPOCH; python3 setup.py bdist_wheel"
1629```
1630
1631### `install_data` / `data_files` problems {#install_data-data_files-problems}
1632
1633If you get the following error:
1634
1635```
1636could not create '/nix/store/6l1bvljpy8gazlsw2aw9skwwp4pmvyxw-python-2.7.8/etc':
1637Permission denied
1638```
1639
1640This is a [known bug](https://github.com/pypa/setuptools/issues/130) in
1641`setuptools`. Setuptools `install_data` does not respect `--prefix`. An example
1642of such package using the feature is `pkgs/tools/X11/xpra/default.nix`.
1643
1644As workaround install it as an extra `preInstall` step:
1645
1646```shell
1647${python.pythonOnBuildForHost.interpreter} setup.py install_data --install-dir=$out --root=$out
1648sed -i '/ = data\_files/d' setup.py
1649```
1650
1651### Rationale of non-existent global site-packages {#rationale-of-non-existent-global-site-packages}
1652
1653On most operating systems a global `site-packages` is maintained. This however
1654becomes problematic if you want to run multiple Python versions or have multiple
1655versions of certain libraries for your projects. Generally, you would solve such
1656issues by creating virtual environments using `virtualenv`.
1657
1658On Nix each package has an isolated dependency tree which, in the case of
1659Python, guarantees the right versions of the interpreter and libraries or
1660packages are available. There is therefore no need to maintain a global `site-packages`.
1661
1662If you want to create a Python environment for development, then the recommended
1663method is to use `nix-shell`, either with or without the [`python.buildEnv`](#python.buildenv-function)
1664function.
1665
1666### How to consume Python modules using pip in a virtual environment like I am used to on other Operating Systems? {#how-to-consume-python-modules-using-pip-in-a-virtual-environment-like-i-am-used-to-on-other-operating-systems}
1667
1668While this approach is not very idiomatic from Nix perspective, it can still be
1669useful when dealing with pre-existing projects or in situations where it's not
1670feasible or desired to write derivations for all required dependencies.
1671
1672This is an example of a `default.nix` for a `nix-shell`, which allows to consume
1673a virtual environment created by `venv`, and install Python modules through
1674`pip` the traditional way.
1675
1676Create this `default.nix` file, together with a `requirements.txt` and
1677execute `nix-shell`.
1678
1679```nix
1680with import <nixpkgs> { };
1681
1682let
1683 pythonPackages = python3Packages;
1684in pkgs.mkShell rec {
1685 name = "impurePythonEnv";
1686 venvDir = "./.venv";
1687 buildInputs = [
1688 # A Python interpreter including the 'venv' module is required to bootstrap
1689 # the environment.
1690 pythonPackages.python
1691
1692 # This executes some shell code to initialize a venv in $venvDir before
1693 # dropping into the shell
1694 pythonPackages.venvShellHook
1695
1696 # Those are dependencies that we would like to use from nixpkgs, which will
1697 # add them to PYTHONPATH and thus make them accessible from within the venv.
1698 pythonPackages.numpy
1699 pythonPackages.requests
1700
1701 # In this particular example, in order to compile any binary extensions they may
1702 # require, the Python modules listed in the hypothetical requirements.txt need
1703 # the following packages to be installed locally:
1704 taglib
1705 openssl
1706 git
1707 libxml2
1708 libxslt
1709 libzip
1710 zlib
1711 ];
1712
1713 # Run this command, only after creating the virtual environment
1714 postVenvCreation = ''
1715 unset SOURCE_DATE_EPOCH
1716 pip install -r requirements.txt
1717 '';
1718
1719 # Now we can execute any commands within the virtual environment.
1720 # This is optional and can be left out to run pip manually.
1721 postShellHook = ''
1722 # allow pip to install wheels
1723 unset SOURCE_DATE_EPOCH
1724 '';
1725
1726}
1727```
1728
1729In case the supplied venvShellHook is insufficient, or when Python 2 support is
1730needed, you can define your own shell hook and adapt to your needs like in the
1731following example:
1732
1733```nix
1734with import <nixpkgs> { };
1735
1736let
1737 venvDir = "./.venv";
1738 pythonPackages = python3Packages;
1739in pkgs.mkShell rec {
1740 name = "impurePythonEnv";
1741 buildInputs = [
1742 pythonPackages.python
1743 # Needed when using python 2.7
1744 # pythonPackages.virtualenv
1745 # ...
1746 ];
1747
1748 # This is very close to how venvShellHook is implemented, but
1749 # adapted to use 'virtualenv'
1750 shellHook = ''
1751 SOURCE_DATE_EPOCH=$(date +%s)
1752
1753 if [ -d "${venvDir}" ]; then
1754 echo "Skipping venv creation, '${venvDir}' already exists"
1755 else
1756 echo "Creating new venv environment in path: '${venvDir}'"
1757 # Note that the module venv was only introduced in python 3, so for 2.7
1758 # this needs to be replaced with a call to virtualenv
1759 ${pythonPackages.python.interpreter} -m venv "${venvDir}"
1760 fi
1761
1762 # Under some circumstances it might be necessary to add your virtual
1763 # environment to PYTHONPATH, which you can do here too;
1764 # PYTHONPATH=$PWD/${venvDir}/${pythonPackages.python.sitePackages}/:$PYTHONPATH
1765
1766 source "${venvDir}/bin/activate"
1767
1768 # As in the previous example, this is optional.
1769 pip install -r requirements.txt
1770 '';
1771}
1772```
1773
1774Note that the `pip install` is an imperative action. So every time `nix-shell`
1775is executed it will attempt to download the Python modules listed in
1776requirements.txt. However these will be cached locally within the `virtualenv`
1777folder and not downloaded again.
1778
1779### How to override a Python package from `configuration.nix`? {#how-to-override-a-python-package-from-configuration.nix}
1780
1781If you need to change a package's attribute(s) from `configuration.nix` you could do:
1782
1783```nix
1784{
1785 nixpkgs.config.packageOverrides = super: {
1786 python3 = super.python3.override {
1787 packageOverrides = python-self: python-super: {
1788 twisted = python-super.twisted.overridePythonAttrs (oldAttrs: {
1789 src = super.fetchPypi {
1790 pname = "Twisted";
1791 version = "19.10.0";
1792 hash = "sha256-c5S6fycq5yKnTz2Wnc9Zm8TvCTvDkgOHSKSQ8XJKUV0=";
1793 extension = "tar.bz2";
1794 };
1795 });
1796 };
1797 };
1798 };
1799}
1800```
1801
1802`python3Packages.twisted` is now globally overridden.
1803All packages and also all NixOS services that reference `twisted`
1804(such as `services.buildbot-worker`) now use the new definition.
1805Note that `python-super` refers to the old package set and `python-self`
1806to the new, overridden version.
1807
1808To modify only a Python package set instead of a whole Python derivation, use
1809this snippet:
1810
1811```nix
1812{
1813 myPythonPackages = python3Packages.override {
1814 overrides = self: super: {
1815 twisted = <...>;
1816 };
1817 };
1818}
1819```
1820
1821### How to override a Python package using overlays? {#how-to-override-a-python-package-using-overlays}
1822
1823Use the following overlay template:
1824
1825```nix
1826self: super: {
1827 python = super.python.override {
1828 packageOverrides = python-self: python-super: {
1829 twisted = python-super.twisted.overrideAttrs (oldAttrs: {
1830 src = super.fetchPypi {
1831 pname = "Twisted";
1832 version = "19.10.0";
1833 hash = "sha256-c5S6fycq5yKnTz2Wnc9Zm8TvCTvDkgOHSKSQ8XJKUV0=";
1834 extension = "tar.bz2";
1835 };
1836 });
1837 };
1838 };
1839}
1840```
1841
1842### How to override a Python package for all Python versions using extensions? {#how-to-override-a-python-package-for-all-python-versions-using-extensions}
1843
1844The following overlay overrides the call to [`buildPythonPackage`](#buildpythonpackage-function) for the
1845`foo` package for all interpreters by appending a Python extension to the
1846`pythonPackagesExtensions` list of extensions.
1847
1848```nix
1849final: prev: {
1850 pythonPackagesExtensions = prev.pythonPackagesExtensions ++ [
1851 (
1852 python-final: python-prev: {
1853 foo = python-prev.foo.overridePythonAttrs (oldAttrs: {
1854 # ...
1855 });
1856 }
1857 )
1858 ];
1859}
1860```
1861
1862### How to use Intel’s MKL with numpy and scipy? {#how-to-use-intels-mkl-with-numpy-and-scipy}
1863
1864MKL can be configured using an overlay. See the section "[Using overlays to
1865configure alternatives](#sec-overlays-alternatives-blas-lapack)".
1866
1867### What inputs do `setup_requires`, `install_requires` and `tests_require` map to? {#what-inputs-do-setup_requires-install_requires-and-tests_require-map-to}
1868
1869In a `setup.py` or `setup.cfg` it is common to declare dependencies:
1870
1871* `setup_requires` corresponds to `build-system`
1872* `install_requires` corresponds to `dependencies`
1873* `tests_require` corresponds to [`nativeCheckInputs`](#var-stdenv-nativeCheckInputs)
1874
1875### How to enable interpreter optimizations? {#optimizations}
1876
1877The Python interpreters are by default not built with optimizations enabled, because
1878the builds are in that case not reproducible. To enable optimizations, override the
1879interpreter of interest, e.g using
1880
1881```nix
1882let
1883 pkgs = import ./. {};
1884 mypython = pkgs.python3.override {
1885 enableOptimizations = true;
1886 reproducibleBuild = false;
1887 self = mypython;
1888 };
1889in mypython
1890```
1891
1892### How to add optional dependencies? {#python-optional-dependencies}
1893
1894Some packages define optional dependencies for additional features. With
1895`setuptools` this is called `extras_require` and `flit` calls it
1896`extras-require`, while PEP 621 calls these `optional-dependencies`.
1897
1898```nix
1899{
1900 optional-dependencies = {
1901 complete = [ distributed ];
1902 };
1903}
1904```
1905
1906and letting the package requiring the extra add the list to its dependencies
1907
1908```nix
1909{
1910 dependencies = [
1911 # ...
1912 ] ++ dask.optional-dependencies.complete;
1913}
1914```
1915
1916This method is using `passthru`, meaning that changing `optional-dependencies` of a package won't cause it to rebuild.
1917
1918Note this method is preferred over adding parameters to builders, as that can
1919result in packages depending on different variants and thereby causing
1920collisions.
1921
1922### How to contribute a Python package to nixpkgs? {#tools}
1923
1924Packages inside nixpkgs must use the [`buildPythonPackage`](#buildpythonpackage-function) or [`buildPythonApplication`](#buildpythonapplication-function) function directly,
1925because we can only provide security support for non-vendored dependencies.
1926
1927We recommend [nix-init](https://github.com/nix-community/nix-init) for creating new python packages within nixpkgs,
1928as it already prefetches the source, parses dependencies for common formats and prefills most things in `meta`.
1929
1930### Are Python interpreters built deterministically? {#deterministic-builds}
1931
1932The Python interpreters are now built deterministically. Minor modifications had
1933to be made to the interpreters in order to generate deterministic bytecode. This
1934has security implications and is relevant for those using Python in a
1935`nix-shell`.
1936
1937When the environment variable `DETERMINISTIC_BUILD` is set, all bytecode will
1938have timestamp 1. The [`buildPythonPackage`](#buildpythonpackage-function) function sets `DETERMINISTIC_BUILD=1`
1939and [PYTHONHASHSEED=0](https://docs.python.org/3.11/using/cmdline.html#envvar-PYTHONHASHSEED).
1940Both are also exported in `nix-shell`.
1941
1942### How to provide automatic tests to Python packages? {#automatic-tests}
1943
1944It is recommended to test packages as part of the build process.
1945Source distributions (`sdist`) often include test files, but not always.
1946
1947By default the command `python setup.py test` is run as part of the
1948[`checkPhase`](#ssec-check-phase), but often it is necessary to pass a custom [`checkPhase`](#ssec-check-phase). An
1949example of such a situation is when `py.test` is used.
1950
1951#### Common issues {#common-issues}
1952
1953* Non-working tests can often be deselected. By default [`buildPythonPackage`](#buildpythonpackage-function)
1954 runs `python setup.py test`. which is deprecated. Most Python modules however
1955 do follow the standard test protocol where the pytest runner can be used
1956 instead. `pytest` supports the `-k` and `--ignore` parameters to ignore test
1957 methods or classes as well as whole files. For `pytestCheckHook` these are
1958 conveniently exposed as `disabledTests` and `disabledTestPaths` respectively.
1959
1960 ```nix
1961 buildPythonPackage {
1962 # ...
1963 nativeCheckInputs = [
1964 pytestCheckHook
1965 ];
1966
1967 disabledTests = [
1968 "function_name"
1969 "other_function"
1970 ];
1971
1972 disabledTestPaths = [
1973 "this/file.py"
1974 ];
1975 }
1976 ```
1977
1978* Tests that attempt to access `$HOME` can be fixed by using the following
1979 work-around before running tests (e.g. `preCheck`): `export HOME=$(mktemp -d)`
1980* Compiling with Cython causes tests to fail with a `ModuleNotLoadedError`.
1981 This can be fixed with two changes in the derivation: 1) replacing `pytest` with
1982 `pytestCheckHook` and 2) adding a `preCheck` containing `cd $out` to run
1983 tests within the built output.
1984
1985## Contributing {#contributing}
1986
1987### Contributing guidelines {#contributing-guidelines}
1988
1989The following rules are desired to be respected:
1990
1991* Python libraries are called from `python-packages.nix` and packaged with
1992 [`buildPythonPackage`](#buildpythonpackage-function). The expression of a library should be in
1993 `pkgs/development/python-modules/<name>/default.nix`.
1994* Python applications live outside of `python-packages.nix` and are packaged
1995 with [`buildPythonApplication`](#buildpythonapplication-function).
1996* Make sure libraries build for all Python interpreters.
1997* By default we enable tests. Make sure the tests are found and, in the case of
1998 libraries, are passing for all interpreters. If certain tests fail they can be
1999 disabled individually. Try to avoid disabling the tests altogether. In any
2000 case, when you disable tests, leave a comment explaining why.
2001* Commit names of Python libraries should reflect that they are Python
2002 libraries, so write for example `python311Packages.numpy: 1.11 -> 1.12`.
2003 It is highly recommended to specify the current default version to enable
2004 automatic build by ofborg.
2005* Attribute names in `python-packages.nix` as well as `pname`s should match the
2006 library's name on PyPI, but be normalized according to [PEP
2007 0503](https://www.python.org/dev/peps/pep-0503/#normalized-names). This means
2008 that characters should be converted to lowercase and `.` and `_` should be
2009 replaced by a single `-` (foo-bar-baz instead of Foo__Bar.baz).
2010 If necessary, `pname` has to be given a different value within `fetchPypi`.
2011* Packages from sources such as GitHub and GitLab that do not exist on PyPI
2012 should not use a name that is already used on PyPI. When possible, they should
2013 use the package repository name prefixed with the owner (e.g. organization) name
2014 and using a `-` as delimiter.
2015* Attribute names in `python-packages.nix` should be sorted alphanumerically to
2016 avoid merge conflicts and ease locating attributes.
2017
2018## Package set maintenance {#python-package-set-maintenance}
2019
2020The whole Python package set has a lot of packages that do not see regular
2021updates, because they either are a very fragile component in the Python
2022ecosystem, like for example the `hypothesis` package, or packages that have
2023no maintainer, so maintenance falls back to the package set maintainers.
2024
2025### Updating packages in bulk {#python-package-bulk-updates}
2026
2027There is a tool to update alot of python libraries in bulk, it exists at
2028`maintainers/scripts/update-python-libraries` with this repository.
2029
2030It can quickly update minor or major versions for all packages selected
2031and create update commits, and supports the `fetchPypi`, `fetchurl` and
2032`fetchFromGitHub` fetchers. When updating lots of packages that are
2033hosted on GitHub, exporting a `GITHUB_API_TOKEN` is highly recommended.
2034
2035Updating packages in bulk leads to lots of breakages, which is why a
2036stabilization period on the `python-updates` branch is required.
2037
2038If a package is fragile and often breaks during these bulks updates, it
2039may be reasonable to set `passthru.skipBulkUpdate = true` in the
2040derivation. This decision should not be made on a whim and should
2041always be supported by a qualifying comment.
2042
2043Once the branch is sufficiently stable it should normally be merged
2044into the `staging` branch.
2045
2046An exemplary call to update all python libraries between minor versions
2047would be:
2048
2049```ShellSession
2050$ maintainers/scripts/update-python-libraries --target minor --commit --use-pkgs-prefix pkgs/development/python-modules/**/default.nix
2051```
2052
2053## CPython Update Schedule {#python-cpython-update-schedule}
2054
2055With [PEP 602](https://www.python.org/dev/peps/pep-0602/), CPython now
2056follows a yearly release cadence. In nixpkgs, all supported interpreters
2057are made available, but only the most recent two
2058interpreters package sets are built; this is a compromise between being
2059the latest interpreter, and what the majority of the Python packages support.
2060
2061New CPython interpreters are released in October. Generally, it takes some
2062time for the majority of active Python projects to support the latest stable
2063interpreter. To help ease the migration for Nixpkgs users
2064between Python interpreters the schedule below will be used:
2065
2066| When | Event |
2067| --- | --- |
2068| After YY.11 Release | Bump CPython package set window. The latest and previous latest stable should now be built. |
2069| After YY.05 Release | Bump default CPython interpreter to latest stable. |
2070
2071In practice, this means that the Python community will have had a stable interpreter
2072for ~2 months before attempting to update the package set. And this will
2073allow for ~7 months for Python applications to support the latest interpreter.