pyrox.dev
1# Python {#python}
2
3## Reference {#reference}
4
5### Interpreters {#interpreters}
6
7| Package | Aliases | Interpreter |
8|------------|-----------------|-------------|
9| python27 | python2, python | CPython 2.7 |
10| python38 | | CPython 3.8 |
11| python39 | | CPython 3.9 |
12| python310 | | CPython 3.10 |
13| python311 | python3 | CPython 3.11 |
14| python312 | | CPython 3.12 |
15| python313 | | CPython 3.13 |
16| pypy27 | pypy2, pypy | PyPy2.7 |
17| pypy39 | pypy3 | PyPy 3.9 |
18
19The Nix expressions for the interpreters can be found in
20`pkgs/development/interpreters/python`.
21
22All packages depending on any Python interpreter get appended
23`out/{python.sitePackages}` to `$PYTHONPATH` if such directory
24exists.
25
26#### Missing `tkinter` module standard library {#missing-tkinter-module-standard-library}
27
28To reduce closure size the `Tkinter`/`tkinter` is available as a separate package, `pythonPackages.tkinter`.
29
30#### Attributes on interpreters packages {#attributes-on-interpreters-packages}
31
32Each interpreter has the following attributes:
33
34- `libPrefix`. Name of the folder in `${python}/lib/` for corresponding interpreter.
35- `interpreter`. Alias for `${python}/bin/${executable}`.
36- `buildEnv`. Function to build python interpreter environments with extra packages bundled together. See [](#python.buildenv-function) for usage and documentation.
37- `withPackages`. Simpler interface to `buildEnv`. See [](#python.withpackages-function) for usage and documentation.
38- `sitePackages`. Alias for `lib/${libPrefix}/site-packages`.
39- `executable`. Name of the interpreter executable, e.g. `python3.10`.
40- `pkgs`. Set of Python packages for that specific interpreter. The package set can be modified by overriding the interpreter and passing `packageOverrides`.
41
42### Building packages and applications {#building-packages-and-applications}
43
44Python libraries and applications that use `setuptools` or
45`distutils` are typically built with respectively the [`buildPythonPackage`](#buildpythonpackage-function) and
46[`buildPythonApplication`](#buildpythonapplication-function) functions. These two functions also support installing a `wheel`.
47
48All Python packages reside in `pkgs/top-level/python-packages.nix` and all
49applications elsewhere. In case a package is used as both a library and an
50application, then the package should be in `pkgs/top-level/python-packages.nix`
51since only those packages are made available for all interpreter versions. The
52preferred location for library expressions is in
53`pkgs/development/python-modules`. It is important that these packages are
54called from `pkgs/top-level/python-packages.nix` and not elsewhere, to guarantee
55the right version of the package is built.
56
57Based on the packages defined in `pkgs/top-level/python-packages.nix` an
58attribute set is created for each available Python interpreter. The available
59sets are
60
61* `pkgs.python27Packages`
62* `pkgs.python3Packages`
63* `pkgs.python38Packages`
64* `pkgs.python39Packages`
65* `pkgs.python310Packages`
66* `pkgs.python311Packages`
67* `pkgs.python312Packages`
68* `pkgs.python313Packages`
69* `pkgs.pypyPackages`
70
71and the aliases
72
73* `pkgs.python2Packages` pointing to `pkgs.python27Packages`
74* `pkgs.python3Packages` pointing to `pkgs.python311Packages`
75* `pkgs.pythonPackages` pointing to `pkgs.python2Packages`
76
77#### `buildPythonPackage` function {#buildpythonpackage-function}
78
79The `buildPythonPackage` function is implemented in
80`pkgs/development/interpreters/python/mk-python-derivation.nix`
81using setup hooks.
82
83The following is an example:
84
85```nix
86{ lib
87, buildPythonPackage
88, fetchPypi
89
90# build-system
91, setuptools-scm
92
93# dependencies
94, attrs
95, pluggy
96, py
97, setuptools
98, six
99
100# tests
101, hypothesis
102 }:
103
104buildPythonPackage rec {
105 pname = "pytest";
106 version = "3.3.1";
107 pyproject = true;
108
109 src = fetchPypi {
110 inherit pname version;
111 hash = "sha256-z4Q23FnYaVNG/NOrKW3kZCXsqwDWQJbOvnn7Ueyy65M=";
112 };
113
114 postPatch = ''
115 # don't test bash builtins
116 rm testing/test_argcomplete.py
117 '';
118
119 nativeBuildInputs = [
120 setuptools-scm
121 ];
122
123 propagatedBuildInputs = [
124 attrs
125 py
126 setuptools
127 six
128 pluggy
129 ];
130
131 nativeCheckInputs = [
132 hypothesis
133 ];
134
135 meta = with lib; {
136 changelog = "https://github.com/pytest-dev/pytest/releases/tag/${version}";
137 description = "Framework for writing tests";
138 homepage = "https://github.com/pytest-dev/pytest";
139 license = licenses.mit;
140 maintainers = with maintainers; [ domenkozar lovek323 madjar lsix ];
141 };
142}
143```
144
145The `buildPythonPackage` mainly does four things:
146
147* In the [`buildPhase`](#build-phase), it calls `${python.pythonOnBuildForHost.interpreter} setup.py bdist_wheel` to
148 build a wheel binary zipfile.
149* In the [`installPhase`](#ssec-install-phase), it installs the wheel file using `pip install *.whl`.
150* In the [`postFixup`](#var-stdenv-postFixup) phase, the `wrapPythonPrograms` bash function is called to
151 wrap all programs in the `$out/bin/*` directory to include `$PATH`
152 environment variable and add dependent libraries to script's `sys.path`.
153* In the [`installCheck`](#ssec-installCheck-phase) phase, `${python.interpreter} setup.py test` is run.
154
155By default tests are run because [`doCheck = true`](#var-stdenv-doCheck). Test dependencies, like
156e.g. the test runner, should be added to [`nativeCheckInputs`](#var-stdenv-nativeCheckInputs).
157
158By default `meta.platforms` is set to the same value
159as the interpreter unless overridden otherwise.
160
161##### `buildPythonPackage` parameters {#buildpythonpackage-parameters}
162
163All parameters from [`stdenv.mkDerivation`](#sec-using-stdenv) function are still supported. The
164following are specific to `buildPythonPackage`:
165
166* `catchConflicts ? true`: If `true`, abort package build if a package name
167 appears more than once in dependency tree. Default is `true`.
168* `disabled ? false`: If `true`, package is not built for the particular Python
169 interpreter version.
170* `dontWrapPythonPrograms ? false`: Skip wrapping of Python programs.
171* `permitUserSite ? false`: Skip setting the `PYTHONNOUSERSITE` environment
172 variable in wrapped programs.
173* `pyproject`: Whether the pyproject format should be used. When set to `true`,
174 `pypaBuildHook` will be used, and you can add the required build dependencies
175 from `build-system.requires` to `nativeBuildInputs`. Note that the pyproject
176 format falls back to using `setuptools`, so you can use `pyproject = true`
177 even if the package only has a `setup.py`. When set to `false`, you can
178 use the existing [hooks](#setup-hooks0 or provide your own logic to build the
179 package. This can be useful for packages that don't support the pyproject
180 format. When unset, the legacy `setuptools` hooks are used for backwards
181 compatibility.
182* `makeWrapperArgs ? []`: A list of strings. Arguments to be passed to
183 [`makeWrapper`](#fun-makeWrapper), which wraps generated binaries. By default, the arguments to
184 [`makeWrapper`](#fun-makeWrapper) set `PATH` and `PYTHONPATH` environment variables before calling
185 the binary. Additional arguments here can allow a developer to set environment
186 variables which will be available when the binary is run. For example,
187 `makeWrapperArgs = ["--set FOO BAR" "--set BAZ QUX"]`.
188* `namePrefix`: Prepends text to `${name}` parameter. In case of libraries, this
189 defaults to `"python3.8-"` for Python 3.8, etc., and in case of applications to `""`.
190* `pipInstallFlags ? []`: A list of strings. Arguments to be passed to `pip
191 install`. To pass options to `python setup.py install`, use
192 `--install-option`. E.g., `pipInstallFlags=["--install-option='--cpp_implementation'"]`.
193* `pipBuildFlags ? []`: A list of strings. Arguments to be passed to `pip wheel`.
194* `pypaBuildFlags ? []`: A list of strings. Arguments to be passed to `python -m build --wheel`.
195* `pythonPath ? []`: List of packages to be added into `$PYTHONPATH`. Packages
196 in `pythonPath` are not propagated (contrary to [`propagatedBuildInputs`](#var-stdenv-propagatedBuildInputs)).
197* `preShellHook`: Hook to execute commands before `shellHook`.
198* `postShellHook`: Hook to execute commands after `shellHook`.
199* `removeBinByteCode ? true`: Remove bytecode from `/bin`. Bytecode is only
200 created when the filenames end with `.py`.
201* `setupPyGlobalFlags ? []`: List of flags passed to `setup.py` command.
202* `setupPyBuildFlags ? []`: List of flags passed to `setup.py build_ext` command.
203
204The [`stdenv.mkDerivation`](#sec-using-stdenv) function accepts various parameters for describing
205build inputs (see "Specifying dependencies"). The following are of special
206interest for Python packages, either because these are primarily used, or
207because their behaviour is different:
208
209* `nativeBuildInputs ? []`: Build-time only dependencies. Typically executables
210 as well as the items listed in `setup_requires`.
211* `buildInputs ? []`: Build and/or run-time dependencies that need to be
212 compiled for the host machine. Typically non-Python libraries which are being
213 linked.
214* `nativeCheckInputs ? []`: Dependencies needed for running the [`checkPhase`](#ssec-check-phase). These
215 are added to [`nativeBuildInputs`](#var-stdenv-nativeBuildInputs) when [`doCheck = true`](#var-stdenv-doCheck). Items listed in
216 `tests_require` go here.
217* `propagatedBuildInputs ? []`: Aside from propagating dependencies,
218 `buildPythonPackage` also injects code into and wraps executables with the
219 paths included in this list. Items listed in `install_requires` go here.
220
221##### Overriding Python packages {#overriding-python-packages}
222
223The `buildPythonPackage` function has a `overridePythonAttrs` method that can be
224used to override the package. In the following example we create an environment
225where we have the `blaze` package using an older version of `pandas`. We
226override first the Python interpreter and pass `packageOverrides` which contains
227the overrides for packages in the package set.
228
229```nix
230with import <nixpkgs> {};
231
232(let
233 python = let
234 packageOverrides = self: super: {
235 pandas = super.pandas.overridePythonAttrs(old: rec {
236 version = "0.19.1";
237 src = fetchPypi {
238 pname = "pandas";
239 inherit version;
240 hash = "sha256-JQn+rtpy/OA2deLszSKEuxyttqBzcAil50H+JDHUdCE=";
241 };
242 });
243 };
244 in pkgs.python3.override {inherit packageOverrides; self = python;};
245
246in python.withPackages(ps: [ ps.blaze ])).env
247```
248
249The next example shows a non trivial overriding of the `blas` implementation to
250be used through out all of the Python package set:
251
252```nix
253python3MyBlas = pkgs.python3.override {
254 packageOverrides = self: super: {
255 # We need toPythonModule for the package set to evaluate this
256 blas = super.toPythonModule(super.pkgs.blas.override {
257 blasProvider = super.pkgs.mkl;
258 });
259 lapack = super.toPythonModule(super.pkgs.lapack.override {
260 lapackProvider = super.pkgs.mkl;
261 });
262 };
263};
264```
265
266This is particularly useful for numpy and scipy users who want to gain speed with other blas implementations.
267Note that using `scipy = super.scipy.override { blas = super.pkgs.mkl; };` will likely result in
268compilation issues, because scipy dependencies need to use the same blas implementation as well.
269
270#### `buildPythonApplication` function {#buildpythonapplication-function}
271
272The [`buildPythonApplication`](#buildpythonapplication-function) function is practically the same as
273[`buildPythonPackage`](#buildpythonpackage-function). The main purpose of this function is to build a Python
274package where one is interested only in the executables, and not importable
275modules. For that reason, when adding this package to a [`python.buildEnv`](#python.buildenv-function), the
276modules won't be made available.
277
278Another difference is that [`buildPythonPackage`](#buildpythonpackage-function) by default prefixes the names of
279the packages with the version of the interpreter. Because this is irrelevant for
280applications, the prefix is omitted.
281
282When packaging a Python application with [`buildPythonApplication`](#buildpythonapplication-function), it should be
283called with `callPackage` and passed `python3` or `python3Packages` (possibly
284specifying an interpreter version), like this:
285
286```nix
287{ lib
288, python3Packages
289, fetchPypi
290}:
291
292python3Packages.buildPythonApplication rec {
293 pname = "luigi";
294 version = "2.7.9";
295 pyproject = true;
296
297 src = fetchPypi {
298 inherit pname version;
299 hash = "sha256-Pe229rT0aHwA98s+nTHQMEFKZPo/yw6sot8MivFDvAw=";
300 };
301
302 nativeBuildInputs = [
303 python3Packages.setuptools
304 python3Packages.wheel
305 ];
306
307 propagatedBuildInputs = [
308 python3Packages.tornado
309 python3Packages.python-daemon
310 ];
311
312 meta = with lib; {
313 # ...
314 };
315}
316```
317
318This is then added to `all-packages.nix` just as any other application would be.
319
320```nix
321luigi = callPackage ../applications/networking/cluster/luigi { };
322```
323
324Since the package is an application, a consumer doesn't need to care about
325Python versions or modules, which is why they don't go in `python3Packages`.
326
327#### `toPythonApplication` function {#topythonapplication-function}
328
329A distinction is made between applications and libraries, however, sometimes a
330package is used as both. In this case the package is added as a library to
331`python-packages.nix` and as an application to `all-packages.nix`. To reduce
332duplication the `toPythonApplication` can be used to convert a library to an
333application.
334
335The Nix expression shall use [`buildPythonPackage`](#buildpythonpackage-function) and be called from
336`python-packages.nix`. A reference shall be created from `all-packages.nix` to
337the attribute in `python-packages.nix`, and the `toPythonApplication` shall be
338applied to the reference:
339
340```nix
341youtube-dl = with python3Packages; toPythonApplication youtube-dl;
342```
343
344#### `toPythonModule` function {#topythonmodule-function}
345
346In some cases, such as bindings, a package is created using
347[`stdenv.mkDerivation`](#sec-using-stdenv) and added as attribute in `all-packages.nix`. The Python
348bindings should be made available from `python-packages.nix`. The
349`toPythonModule` function takes a derivation and makes certain Python-specific
350modifications.
351
352```nix
353opencv = toPythonModule (pkgs.opencv.override {
354 enablePython = true;
355 pythonPackages = self;
356});
357```
358
359Do pay attention to passing in the right Python version!
360
361#### `python.buildEnv` function {#python.buildenv-function}
362
363Python environments can be created using the low-level `pkgs.buildEnv` function.
364This example shows how to create an environment that has the Pyramid Web Framework.
365Saving the following as `default.nix`
366
367```nix
368with import <nixpkgs> {};
369
370python3.buildEnv.override {
371 extraLibs = [ python3Packages.pyramid ];
372 ignoreCollisions = true;
373}
374```
375
376and running `nix-build` will create
377
378```
379/nix/store/cf1xhjwzmdki7fasgr4kz6di72ykicl5-python-2.7.8-env
380```
381
382with wrapped binaries in `bin/`.
383
384You can also use the `env` attribute to create local environments with needed
385packages installed. This is somewhat comparable to `virtualenv`. For example,
386running `nix-shell` with the following `shell.nix`
387
388```nix
389with import <nixpkgs> {};
390
391(python3.buildEnv.override {
392 extraLibs = with python3Packages; [
393 numpy
394 requests
395 ];
396}).env
397```
398
399will drop you into a shell where Python will have the
400specified packages in its path.
401
402##### `python.buildEnv` arguments {#python.buildenv-arguments}
403
404
405* `extraLibs`: List of packages installed inside the environment.
406* `postBuild`: Shell command executed after the build of environment.
407* `ignoreCollisions`: Ignore file collisions inside the environment (default is `false`).
408* `permitUserSite`: Skip setting the `PYTHONNOUSERSITE` environment variable in
409 wrapped binaries in the environment.
410
411#### `python.withPackages` function {#python.withpackages-function}
412
413The [`python.withPackages`](#python.withpackages-function) function provides a simpler interface to the [`python.buildEnv`](#python.buildenv-function) functionality.
414It takes a function as an argument that is passed the set of python packages and returns the list
415of the packages to be included in the environment. Using the [`withPackages`](#python.withpackages-function) function, the previous
416example for the Pyramid Web Framework environment can be written like this:
417
418```nix
419with import <nixpkgs> {};
420
421python.withPackages (ps: [ ps.pyramid ])
422```
423
424[`withPackages`](#python.withpackages-function) passes the correct package set for the specific interpreter
425version as an argument to the function. In the above example, `ps` equals
426`pythonPackages`. But you can also easily switch to using python3:
427
428```nix
429with import <nixpkgs> {};
430
431python3.withPackages (ps: [ ps.pyramid ])
432```
433
434Now, `ps` is set to `python3Packages`, matching the version of the interpreter.
435
436As [`python.withPackages`](#python.withpackages-function) uses [`python.buildEnv`](#python.buildenv-function) under the hood, it also
437supports the `env` attribute. The `shell.nix` file from the previous section can
438thus be also written like this:
439
440```nix
441with import <nixpkgs> {};
442
443(python3.withPackages (ps: with ps; [
444 numpy
445 requests
446])).env
447```
448
449In contrast to [`python.buildEnv`](#python.buildenv-function), [`python.withPackages`](#python.withpackages-function) does not support the
450more advanced options such as `ignoreCollisions = true` or `postBuild`. If you
451need them, you have to use [`python.buildEnv`](#python.buildenv-function).
452
453Python 2 namespace packages may provide `__init__.py` that collide. In that case
454[`python.buildEnv`](#python.buildenv-function) should be used with `ignoreCollisions = true`.
455
456#### Setup hooks {#setup-hooks}
457
458The following are setup hooks specifically for Python packages. Most of these
459are used in [`buildPythonPackage`](#buildpythonpackage-function).
460
461- `eggUnpackhook` to move an egg to the correct folder so it can be installed
462 with the `eggInstallHook`
463- `eggBuildHook` to skip building for eggs.
464- `eggInstallHook` to install eggs.
465- `pipBuildHook` to build a wheel using `pip` and PEP 517. Note a build system
466 (e.g. `setuptools` or `flit`) should still be added as `nativeBuildInput`.
467- `pypaBuildHook` to build a wheel using
468 [`pypa/build`](https://pypa-build.readthedocs.io/en/latest/index.html) and
469 PEP 517/518. Note a build system (e.g. `setuptools` or `flit`) should still
470 be added as `nativeBuildInput`.
471- `pipInstallHook` to install wheels.
472- `pytestCheckHook` to run tests with `pytest`. See [example usage](#using-pytestcheckhook).
473- `pythonCatchConflictsHook` to check whether a Python package is not already existing.
474- `pythonImportsCheckHook` to check whether importing the listed modules works.
475- `pythonRelaxDepsHook` will relax Python dependencies restrictions for the package.
476 See [example usage](#using-pythonrelaxdepshook).
477- `pythonRemoveBinBytecode` to remove bytecode from the `/bin` folder.
478- `setuptoolsBuildHook` to build a wheel using `setuptools`.
479- `setuptoolsCheckHook` to run tests with `python setup.py test`.
480- `sphinxHook` to build documentation and manpages using Sphinx.
481- `venvShellHook` to source a Python 3 `venv` at the `venvDir` location. A
482 `venv` is created if it does not yet exist. `postVenvCreation` can be used to
483 to run commands only after venv is first created.
484- `wheelUnpackHook` to move a wheel to the correct folder so it can be installed
485 with the `pipInstallHook`.
486- `unittestCheckHook` will run tests with `python -m unittest discover`. See [example usage](#using-unittestcheckhook).
487
488### Development mode {#development-mode}
489
490Development or editable mode is supported. To develop Python packages
491[`buildPythonPackage`](#buildpythonpackage-function) has additional logic inside `shellPhase` to run `pip
492install -e . --prefix $TMPDIR/`for the package.
493
494Warning: `shellPhase` is executed only if `setup.py` exists.
495
496Given a `default.nix`:
497
498```nix
499with import <nixpkgs> {};
500
501python3Packages.buildPythonPackage {
502 name = "myproject";
503 buildInputs = with python3Packages; [ pyramid ];
504
505 src = ./.;
506}
507```
508
509Running `nix-shell` with no arguments should give you the environment in which
510the package would be built with `nix-build`.
511
512Shortcut to setup environments with C headers/libraries and Python packages:
513
514```shell
515nix-shell -p python3Packages.pyramid zlib libjpeg git
516```
517
518::: {.note}
519There is a boolean value `lib.inNixShell` set to `true` if nix-shell is invoked.
520:::
521
522## User Guide {#user-guide}
523
524### Using Python {#using-python}
525
526#### Overview {#overview}
527
528Several versions of the Python interpreter are available on Nix, as well as a
529high amount of packages. The attribute `python3` refers to the default
530interpreter, which is currently CPython 3.11. The attribute `python` refers to
531CPython 2.7 for backwards-compatibility. It is also possible to refer to
532specific versions, e.g. `python311` refers to CPython 3.11, and `pypy` refers to
533the default PyPy interpreter.
534
535Python is used a lot, and in different ways. This affects also how it is
536packaged. In the case of Python on Nix, an important distinction is made between
537whether the package is considered primarily an application, or whether it should
538be used as a library, i.e., of primary interest are the modules in
539`site-packages` that should be importable.
540
541In the Nixpkgs tree Python applications can be found throughout, depending on
542what they do, and are called from the main package set. Python libraries,
543however, are in separate sets, with one set per interpreter version.
544
545The interpreters have several common attributes. One of these attributes is
546`pkgs`, which is a package set of Python libraries for this specific
547interpreter. E.g., the `toolz` package corresponding to the default interpreter
548is `python3.pkgs.toolz`, and the CPython 3.11 version is `python311.pkgs.toolz`.
549The main package set contains aliases to these package sets, e.g.
550`pythonPackages` refers to `python.pkgs` and `python311Packages` to
551`python311.pkgs`.
552
553#### Installing Python and packages {#installing-python-and-packages}
554
555The Nix and NixOS manuals explain how packages are generally installed. In the
556case of Python and Nix, it is important to make a distinction between whether the
557package is considered an application or a library.
558
559Applications on Nix are typically installed into your user profile imperatively
560using `nix-env -i`, and on NixOS declaratively by adding the package name to
561`environment.systemPackages` in `/etc/nixos/configuration.nix`. Dependencies
562such as libraries are automatically installed and should not be installed
563explicitly.
564
565The same goes for Python applications. Python applications can be installed in
566your profile, and will be wrapped to find their exact library dependencies,
567without impacting other applications or polluting your user environment.
568
569But Python libraries you would like to use for development cannot be installed,
570at least not individually, because they won't be able to find each other
571resulting in import errors. Instead, it is possible to create an environment
572with [`python.buildEnv`](#python.buildenv-function) or [`python.withPackages`](#python.withpackages-function) where the interpreter and other
573executables are wrapped to be able to find each other and all of the modules.
574
575In the following examples we will start by creating a simple, ad-hoc environment
576with a nix-shell that has `numpy` and `toolz` in Python 3.11; then we will create
577a re-usable environment in a single-file Python script; then we will create a
578full Python environment for development with this same environment.
579
580Philosophically, this should be familiar to users who are used to a `venv` style
581of development: individual projects create their own Python environments without
582impacting the global environment or each other.
583
584#### Ad-hoc temporary Python environment with `nix-shell` {#ad-hoc-temporary-python-environment-with-nix-shell}
585
586The simplest way to start playing with the way nix wraps and sets up Python
587environments is with `nix-shell` at the cmdline. These environments create a
588temporary shell session with a Python and a *precise* list of packages (plus
589their runtime dependencies), with no other Python packages in the Python
590interpreter's scope.
591
592To create a Python 3.11 session with `numpy` and `toolz` available, run:
593
594```sh
595$ nix-shell -p 'python311.withPackages(ps: with ps; [ numpy toolz ])'
596```
597
598By default `nix-shell` will start a `bash` session with this interpreter in our
599`PATH`, so if we then run:
600
601```Python console
602[nix-shell:~/src/nixpkgs]$ python3
603Python 3.11.3 (main, Apr 4 2023, 22:36:41) [GCC 12.2.0] on linux
604Type "help", "copyright", "credits" or "license" for more information.
605>>> import numpy; import toolz
606```
607
608Note that no other modules are in scope, even if they were imperatively
609installed into our user environment as a dependency of a Python application:
610
611```Python console
612>>> import requests
613Traceback (most recent call last):
614 File "<stdin>", line 1, in <module>
615ModuleNotFoundError: No module named 'requests'
616```
617
618We can add as many additional modules onto the `nix-shell` as we need, and we
619will still get 1 wrapped Python interpreter. We can start the interpreter
620directly like so:
621
622```sh
623$ nix-shell -p "python311.withPackages (ps: with ps; [ numpy toolz requests ])" --run python3
624this derivation will be built:
625 /nix/store/r19yf5qgfiakqlhkgjahbg3zg79549n4-python3-3.11.2-env.drv
626building '/nix/store/r19yf5qgfiakqlhkgjahbg3zg79549n4-python3-3.11.2-env.drv'...
627created 273 symlinks in user environment
628Python 3.11.2 (main, Feb 7 2023, 13:52:42) [GCC 12.2.0] on linux
629Type "help", "copyright", "credits" or "license" for more information.
630>>> import requests
631>>>
632```
633
634Notice that this time it built a new Python environment, which now includes
635`requests`. Building an environment just creates wrapper scripts that expose the
636selected dependencies to the interpreter while re-using the actual modules. This
637means if any other env has installed `requests` or `numpy` in a different
638context, we don't need to recompile them -- we just recompile the wrapper script
639that sets up an interpreter pointing to them. This matters much more for "big"
640modules like `pytorch` or `tensorflow`.
641
642Module names usually match their names on [pypi.org](https://pypi.org/), but
643you can use the [Nixpkgs search website](https://nixos.org/nixos/packages.html)
644to find them as well (along with non-python packages).
645
646At this point we can create throwaway experimental Python environments with
647arbitrary dependencies. This is a good way to get a feel for how the Python
648interpreter and dependencies work in Nix and NixOS, but to do some actual
649development, we'll want to make it a bit more persistent.
650
651##### Running Python scripts and using `nix-shell` as shebang {#running-python-scripts-and-using-nix-shell-as-shebang}
652
653Sometimes, we have a script whose header looks like this:
654
655```python
656#!/usr/bin/env python3
657import numpy as np
658a = np.array([1,2])
659b = np.array([3,4])
660print(f"The dot product of {a} and {b} is: {np.dot(a, b)}")
661```
662
663Executing this script requires a `python3` that has `numpy`. Using what we learned
664in the previous section, we could startup a shell and just run it like so:
665
666```ShellSession
667$ nix-shell -p 'python311.withPackages (ps: with ps; [ numpy ])' --run 'python3 foo.py'
668The dot product of [1 2] and [3 4] is: 11
669```
670
671But if we maintain the script ourselves, and if there are more dependencies, it
672may be nice to encode those dependencies in source to make the script re-usable
673without that bit of knowledge. That can be done by using `nix-shell` as a
674[shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)), like so:
675
676```python
677#!/usr/bin/env nix-shell
678#!nix-shell -i python3 -p "python3.withPackages(ps: [ ps.numpy ])"
679import numpy as np
680a = np.array([1,2])
681b = np.array([3,4])
682print(f"The dot product of {a} and {b} is: {np.dot(a, b)}")
683```
684
685Then we execute it, without requiring any environment setup at all!
686
687```sh
688$ ./foo.py
689The dot product of [1 2] and [3 4] is: 11
690```
691
692If the dependencies are not available on the host where `foo.py` is executed, it
693will build or download them from a Nix binary cache prior to starting up, prior
694that it is executed on a machine with a multi-user nix installation.
695
696This provides a way to ship a self bootstrapping Python script, akin to a
697statically linked binary, where it can be run on any machine (provided nix is
698installed) without having to assume that `numpy` is installed globally on the
699system.
700
701By default it is pulling the import checkout of Nixpkgs itself from our nix
702channel, which is nice as it cache aligns with our other package builds, but we
703can make it fully reproducible by pinning the `nixpkgs` import:
704
705```python
706#!/usr/bin/env nix-shell
707#!nix-shell -i python3 -p "python3.withPackages (ps: [ ps.numpy ])"
708#!nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/e51209796c4262bfb8908e3d6d72302fe4e96f5f.tar.gz
709import numpy as np
710a = np.array([1,2])
711b = np.array([3,4])
712print(f"The dot product of {a} and {b} is: {np.dot(a, b)}")
713```
714
715This will execute with the exact same versions of Python 3.10, numpy, and system
716dependencies a year from now as it does today, because it will always use
717exactly git commit `e51209796c4262bfb8908e3d6d72302fe4e96f5f` of Nixpkgs for all
718of the package versions.
719
720This is also a great way to ensure the script executes identically on different
721servers.
722
723##### Load environment from `.nix` expression {#load-environment-from-.nix-expression}
724
725We've now seen how to create an ad-hoc temporary shell session, and how to
726create a single script with Python dependencies, but in the course of normal
727development we're usually working in an entire package repository.
728
729As explained [in the `nix-shell` section](https://nixos.org/manual/nix/stable/command-ref/nix-shell) of the Nix manual, `nix-shell` can also load an expression from a `.nix` file.
730Say we want to have Python 3.11, `numpy` and `toolz`, like before,
731in an environment. We can add a `shell.nix` file describing our dependencies:
732
733```nix
734with import <nixpkgs> {};
735(python311.withPackages (ps: with ps; [
736 numpy
737 toolz
738])).env
739```
740
741And then at the command line, just typing `nix-shell` produces the same
742environment as before. In a normal project, we'll likely have many more
743dependencies; this can provide a way for developers to share the environments
744with each other and with CI builders.
745
746What's happening here?
747
7481. We begin with importing the Nix Packages collections. `import <nixpkgs>`
749 imports the `<nixpkgs>` function, `{}` calls it and the `with` statement
750 brings all attributes of `nixpkgs` in the local scope. These attributes form
751 the main package set.
7522. Then we create a Python 3.11 environment with the [`withPackages`](#python.withpackages-function) function, as before.
7533. The [`withPackages`](#python.withpackages-function) function expects us to provide a function as an argument
754 that takes the set of all Python packages and returns a list of packages to
755 include in the environment. Here, we select the packages `numpy` and `toolz`
756 from the package set.
757
758To combine this with `mkShell` you can:
759
760```nix
761with import <nixpkgs> {};
762let
763 pythonEnv = python311.withPackages (ps: [
764 ps.numpy
765 ps.toolz
766 ]);
767in mkShell {
768 packages = [
769 pythonEnv
770
771 black
772 mypy
773
774 libffi
775 openssl
776 ];
777}
778```
779
780This will create a unified environment that has not just our Python interpreter
781and its Python dependencies, but also tools like `black` or `mypy` and libraries
782like `libffi` the `openssl` in scope. This is generic and can span any number of
783tools or languages across the Nixpkgs ecosystem.
784
785##### Installing environments globally on the system {#installing-environments-globally-on-the-system}
786
787Up to now, we've been creating environments scoped to an ad-hoc shell session,
788or a single script, or a single project. This is generally advisable, as it
789avoids pollution across contexts.
790
791However, sometimes we know we will often want a Python with some basic packages,
792and want this available without having to enter into a shell or build context.
793This can be useful to have things like vim/emacs editors and plugins or shell
794tools "just work" without having to set them up, or when running other software
795that expects packages to be installed globally.
796
797To create your own custom environment, create a file in `~/.config/nixpkgs/overlays/`
798that looks like this:
799
800```nix
801# ~/.config/nixpkgs/overlays/myEnv.nix
802self: super: {
803 myEnv = super.buildEnv {
804 name = "myEnv";
805 paths = [
806 # A Python 3 interpreter with some packages
807 (self.python3.withPackages (
808 ps: with ps; [
809 pyflakes
810 pytest
811 black
812 ]
813 ))
814
815 # Some other packages we'd like as part of this env
816 self.mypy
817 self.black
818 self.ripgrep
819 self.tmux
820 ];
821 };
822}
823```
824
825You can then build and install this to your profile with:
826
827```sh
828nix-env -iA myEnv
829```
830
831One limitation of this is that you can only have 1 Python env installed
832globally, since they conflict on the `python` to load out of your `PATH`.
833
834If you get a conflict or prefer to keep the setup clean, you can have `nix-env`
835atomically *uninstall* all other imperatively installed packages and replace
836your profile with just `myEnv` by using the `--replace` flag.
837
838##### Environment defined in `/etc/nixos/configuration.nix` {#environment-defined-in-etcnixosconfiguration.nix}
839
840For the sake of completeness, here's how to install the environment system-wide
841on NixOS.
842
843```nix
844{ # ...
845
846 environment.systemPackages = with pkgs; [
847 (python310.withPackages(ps: with ps; [ numpy toolz ]))
848 ];
849}
850```
851
852### Developing with Python {#developing-with-python}
853
854Above, we were mostly just focused on use cases and what to do to get started
855creating working Python environments in nix.
856
857Now that you know the basics to be up and running, it is time to take a step
858back and take a deeper look at how Python packages are packaged on Nix. Then,
859we will look at how you can use development mode with your code.
860
861#### Python library packages in Nixpkgs {#python-library-packages-in-nixpkgs}
862
863With Nix all packages are built by functions. The main function in Nix for
864building Python libraries is [`buildPythonPackage`](#buildpythonpackage-function). Let's see how we can build the
865`toolz` package.
866
867```nix
868{ lib
869, buildPythonPackage
870, fetchPypi
871, setuptools
872, wheel
873}:
874
875buildPythonPackage rec {
876 pname = "toolz";
877 version = "0.10.0";
878 pyproject = true;
879
880 src = fetchPypi {
881 inherit pname version;
882 hash = "sha256-CP3V73yWSArRHBLUct4hrNMjWZlvaaUlkpm1QP66RWA=";
883 };
884
885 nativeBuildInputs = [
886 setuptools
887 wheel
888 ];
889
890 # has no tests
891 doCheck = false;
892
893 pythonImportsCheck = [
894 "toolz.itertoolz"
895 "toolz.functoolz"
896 "toolz.dicttoolz"
897 ];
898
899 meta = with lib; {
900 changelog = "https://github.com/pytoolz/toolz/releases/tag/${version}";
901 homepage = "https://github.com/pytoolz/toolz";
902 description = "List processing tools and functional utilities";
903 license = licenses.bsd3;
904 maintainers = with maintainers; [ fridh ];
905 };
906}
907```
908
909What happens here? The function [`buildPythonPackage`](#buildpythonpackage-function) is called and as argument
910it accepts a set. In this case the set is a recursive set, `rec`. One of the
911arguments is the name of the package, which consists of a basename (generally
912following the name on PyPi) and a version. Another argument, `src` specifies the
913source, which in this case is fetched from PyPI using the helper function
914`fetchPypi`. The argument `doCheck` is used to set whether tests should be run
915when building the package. Since there are no tests, we rely on [`pythonImportsCheck`](#using-pythonimportscheck)
916to test whether the package can be imported. Furthermore, we specify some meta
917information. The output of the function is a derivation.
918
919An expression for `toolz` can be found in the Nixpkgs repository. As explained
920in the introduction of this Python section, a derivation of `toolz` is available
921for each interpreter version, e.g. `python311.pkgs.toolz` refers to the `toolz`
922derivation corresponding to the CPython 3.11 interpreter.
923
924The above example works when you're directly working on
925`pkgs/top-level/python-packages.nix` in the Nixpkgs repository. Often though,
926you will want to test a Nix expression outside of the Nixpkgs tree.
927
928The following expression creates a derivation for the `toolz` package,
929and adds it along with a `numpy` package to a Python environment.
930
931```nix
932with import <nixpkgs> {};
933
934( let
935 my_toolz = python311.pkgs.buildPythonPackage rec {
936 pname = "toolz";
937 version = "0.10.0";
938 pyproject = true;
939
940 src = fetchPypi {
941 inherit pname version;
942 hash = "sha256-CP3V73yWSArRHBLUct4hrNMjWZlvaaUlkpm1QP66RWA=";
943 };
944
945 nativeBuildInputs = [
946 python311.pkgs.setuptools
947 python311.pkgs.wheel
948 ];
949
950 # has no tests
951 doCheck = false;
952
953 meta = {
954 homepage = "https://github.com/pytoolz/toolz/";
955 description = "List processing tools and functional utilities";
956 # [...]
957 };
958 };
959
960 in python311.withPackages (ps: with ps; [
961 numpy
962 my_toolz
963 ])
964).env
965```
966
967Executing `nix-shell` will result in an environment in which you can use
968Python 3.11 and the `toolz` package. As you can see we had to explicitly mention
969for which Python version we want to build a package.
970
971So, what did we do here? Well, we took the Nix expression that we used earlier
972to build a Python environment, and said that we wanted to include our own
973version of `toolz`, named `my_toolz`. To introduce our own package in the scope
974of [`withPackages`](#python.withpackages-function) we used a `let` expression. You can see that we used
975`ps.numpy` to select numpy from the nixpkgs package set (`ps`). We did not take
976`toolz` from the Nixpkgs package set this time, but instead took our own version
977that we introduced with the `let` expression.
978
979#### Handling dependencies {#handling-dependencies}
980
981Our example, `toolz`, does not have any dependencies on other Python packages or
982system libraries. According to the manual, [`buildPythonPackage`](#buildpythonpackage-function) uses the
983arguments [`buildInputs`](#var-stdenv-buildInputs) and [`propagatedBuildInputs`](#var-stdenv-propagatedBuildInputs) to specify dependencies. If
984something is exclusively a build-time dependency, then the dependency should be
985included in [`buildInputs`](#var-stdenv-buildInputs), but if it is (also) a runtime dependency, then it
986should be added to [`propagatedBuildInputs`](#var-stdenv-propagatedBuildInputs). Test dependencies are considered
987build-time dependencies and passed to [`nativeCheckInputs`](#var-stdenv-nativeCheckInputs).
988
989The following example shows which arguments are given to [`buildPythonPackage`](#buildpythonpackage-function) in
990order to build [`datashape`](https://github.com/blaze/datashape).
991
992```nix
993{ lib
994, buildPythonPackage
995, fetchPypi
996
997# build dependencies
998, setuptools, wheel
999
1000# dependencies
1001, numpy, multipledispatch, python-dateutil
1002
1003# tests
1004, pytest
1005}:
1006
1007buildPythonPackage rec {
1008 pname = "datashape";
1009 version = "0.4.7";
1010 pyproject = true;
1011
1012 src = fetchPypi {
1013 inherit pname version;
1014 hash = "sha256-FLLvdm1MllKrgTGC6Gb0k0deZeVYvtCCLji/B7uhong=";
1015 };
1016
1017 nativeBuildInputs = [
1018 setuptools
1019 wheel
1020 ];
1021
1022 propagatedBuildInputs = [
1023 multipledispatch
1024 numpy
1025 python-dateutil
1026 ];
1027
1028 nativeCheckInputs = [
1029 pytest
1030 ];
1031
1032 meta = with lib; {
1033 changelog = "https://github.com/blaze/datashape/releases/tag/${version}";
1034 homepage = "https://github.com/ContinuumIO/datashape";
1035 description = "A data description language";
1036 license = licenses.bsd2;
1037 maintainers = with maintainers; [ fridh ];
1038 };
1039}
1040```
1041
1042We can see several runtime dependencies, `numpy`, `multipledispatch`, and
1043`python-dateutil`. Furthermore, we have [`nativeCheckInputs`](#var-stdenv-nativeCheckInputs) with `pytest`.
1044`pytest` is a test runner and is only used during the [`checkPhase`](#ssec-check-phase) and is
1045therefore not added to [`propagatedBuildInputs`](#var-stdenv-propagatedBuildInputs).
1046
1047In the previous case we had only dependencies on other Python packages to consider.
1048Occasionally you have also system libraries to consider. E.g., `lxml` provides
1049Python bindings to `libxml2` and `libxslt`. These libraries are only required
1050when building the bindings and are therefore added as [`buildInputs`](#var-stdenv-buildInputs).
1051
1052```nix
1053{ lib
1054, buildPythonPackage
1055, fetchPypi
1056, setuptools
1057, wheel
1058, libxml2
1059, libxslt
1060}:
1061
1062buildPythonPackage rec {
1063 pname = "lxml";
1064 version = "3.4.4";
1065 pyproject = true;
1066
1067 src = fetchPypi {
1068 inherit pname version;
1069 hash = "sha256-s9NiusRxFydHzaNRMjjxFcvWxfi45jGb9ql6eJJyQJk=";
1070 };
1071
1072 nativeBuildInputs = [
1073 setuptools
1074 wheel
1075 ];
1076
1077 buildInputs = [
1078 libxml2
1079 libxslt
1080 ];
1081
1082 meta = with lib; {
1083 changelog = "https://github.com/lxml/lxml/releases/tag/lxml-${version}";
1084 description = "Pythonic binding for the libxml2 and libxslt libraries";
1085 homepage = "https://lxml.de";
1086 license = licenses.bsd3;
1087 maintainers = with maintainers; [ sjourdois ];
1088 };
1089}
1090```
1091
1092In this example `lxml` and Nix are able to work out exactly where the relevant
1093files of the dependencies are. This is not always the case.
1094
1095The example below shows bindings to The Fastest Fourier Transform in the West,
1096commonly known as FFTW. On Nix we have separate packages of FFTW for the
1097different types of floats (`"single"`, `"double"`, `"long-double"`). The
1098bindings need all three types, and therefore we add all three as [`buildInputs`](#var-stdenv-buildInputs).
1099The bindings don't expect to find each of them in a different folder, and
1100therefore we have to set `LDFLAGS` and `CFLAGS`.
1101
1102```nix
1103{ lib
1104, buildPythonPackage
1105, fetchPypi
1106
1107# build dependencies
1108, setuptools
1109, wheel
1110
1111# dependencies
1112, fftw
1113, fftwFloat
1114, fftwLongDouble
1115, numpy
1116, scipy
1117}:
1118
1119buildPythonPackage rec {
1120 pname = "pyFFTW";
1121 version = "0.9.2";
1122 pyproject = true;
1123
1124 src = fetchPypi {
1125 inherit pname version;
1126 hash = "sha256-9ru2r6kwhUCaskiFoaPNuJCfCVoUL01J40byvRt4kHQ=";
1127 };
1128
1129 nativeBuildInputs = [
1130 setuptools
1131 wheel
1132 ];
1133
1134 buildInputs = [
1135 fftw
1136 fftwFloat
1137 fftwLongDouble
1138 ];
1139
1140 propagatedBuildInputs = [
1141 numpy
1142 scipy
1143 ];
1144
1145 preConfigure = ''
1146 export LDFLAGS="-L${fftw.dev}/lib -L${fftwFloat.out}/lib -L${fftwLongDouble.out}/lib"
1147 export CFLAGS="-I${fftw.dev}/include -I${fftwFloat.dev}/include -I${fftwLongDouble.dev}/include"
1148 '';
1149
1150 # Tests cannot import pyfftw. pyfftw works fine though.
1151 doCheck = false;
1152
1153 meta = with lib; {
1154 changelog = "https://github.com/pyFFTW/pyFFTW/releases/tag/v${version}";
1155 description = "A pythonic wrapper around FFTW, the FFT library, presenting a unified interface for all the supported transforms";
1156 homepage = "http://hgomersall.github.com/pyFFTW";
1157 license = with licenses; [ bsd2 bsd3 ];
1158 maintainers = with maintainers; [ fridh ];
1159 };
1160}
1161```
1162
1163Note also the line [`doCheck = false;`](#var-stdenv-doCheck), we explicitly disabled running the test-suite.
1164
1165#### Testing Python Packages {#testing-python-packages}
1166
1167It is highly encouraged to have testing as part of the package build. This
1168helps to avoid situations where the package was able to build and install,
1169but is not usable at runtime. Currently, all packages will use the `test`
1170command provided by the setup.py (i.e. `python setup.py test`). However,
1171this is currently deprecated https://github.com/pypa/setuptools/pull/1878
1172and your package should provide its own [`checkPhase`](#ssec-check-phase).
1173
1174::: {.note}
1175The [`checkPhase`](#ssec-check-phase) for python maps to the `installCheckPhase` on a
1176normal derivation. This is due to many python packages not behaving well
1177to the pre-installed version of the package. Version info, and natively
1178compiled extensions generally only exist in the install directory, and
1179thus can cause issues when a test suite asserts on that behavior.
1180:::
1181
1182::: {.note}
1183Tests should only be disabled if they don't agree with nix
1184(e.g. external dependencies, network access, flakey tests), however,
1185as many tests should be enabled as possible. Failing tests can still be
1186a good indication that the package is not in a valid state.
1187:::
1188
1189#### Using pytest {#using-pytest}
1190
1191Pytest is the most common test runner for python repositories. A trivial
1192test run would be:
1193
1194```
1195 nativeCheckInputs = [ pytest ];
1196 checkPhase = ''
1197 runHook preCheck
1198
1199 pytest
1200
1201 runHook postCheck
1202 '';
1203```
1204
1205However, many repositories' test suites do not translate well to nix's build
1206sandbox, and will generally need many tests to be disabled.
1207
1208To filter tests using pytest, one can do the following:
1209
1210```
1211 nativeCheckInputs = [ pytest ];
1212 # avoid tests which need additional data or touch network
1213 checkPhase = ''
1214 runHook preCheck
1215
1216 pytest tests/ --ignore=tests/integration -k 'not download and not update' --ignore=tests/test_failing.py
1217
1218 runHook postCheck
1219 '';
1220```
1221
1222`--ignore` will tell pytest to ignore that file or directory from being
1223collected as part of a test run. This is useful is a file uses a package
1224which is not available in nixpkgs, thus skipping that test file is much
1225easier than having to create a new package.
1226
1227`-k` is used to define a predicate for test names. In this example, we are
1228filtering out tests which contain `download` or `update` in their test case name.
1229Only one `-k` argument is allowed, and thus a long predicate should be concatenated
1230with “\\” and wrapped to the next line.
1231
1232::: {.note}
1233In pytest==6.0.1, the use of “\\” to continue a line (e.g. `-k 'not download \'`) has
1234been removed, in this case, it's recommended to use `pytestCheckHook`.
1235:::
1236
1237#### Using pytestCheckHook {#using-pytestcheckhook}
1238
1239`pytestCheckHook` is a convenient hook which will substitute the setuptools
1240`test` command for a [`checkPhase`](#ssec-check-phase) which runs `pytest`. This is also beneficial
1241when a package may need many items disabled to run the test suite.
1242
1243Using the example above, the analogous `pytestCheckHook` usage would be:
1244
1245```
1246 nativeCheckInputs = [
1247 pytestCheckHook
1248 ];
1249
1250 # requires additional data
1251 pytestFlagsArray = [
1252 "tests/"
1253 "--ignore=tests/integration"
1254 ];
1255
1256 disabledTests = [
1257 # touches network
1258 "download"
1259 "update"
1260 ];
1261
1262 disabledTestPaths = [
1263 "tests/test_failing.py"
1264 ];
1265```
1266
1267This is especially useful when tests need to be conditionally disabled,
1268for example:
1269
1270```
1271 disabledTests = [
1272 # touches network
1273 "download"
1274 "update"
1275 ] ++ lib.optionals (pythonAtLeast "3.8") [
1276 # broken due to python3.8 async changes
1277 "async"
1278 ] ++ lib.optionals stdenv.isDarwin [
1279 # can fail when building with other packages
1280 "socket"
1281 ];
1282```
1283
1284Trying to concatenate the related strings to disable tests in a regular
1285[`checkPhase`](#ssec-check-phase) would be much harder to read. This also enables us to comment on
1286why specific tests are disabled.
1287
1288#### Using pythonImportsCheck {#using-pythonimportscheck}
1289
1290Although unit tests are highly preferred to validate correctness of a package, not
1291all packages have test suites that can be run easily, and some have none at all.
1292To help ensure the package still works, [`pythonImportsCheck`](#using-pythonimportscheck) can attempt to import
1293the listed modules.
1294
1295```
1296 pythonImportsCheck = [
1297 "requests"
1298 "urllib"
1299 ];
1300```
1301
1302roughly translates to:
1303
1304```
1305 postCheck = ''
1306 PYTHONPATH=$out/${python.sitePackages}:$PYTHONPATH
1307 python -c "import requests; import urllib"
1308 '';
1309```
1310
1311However, this is done in its own phase, and not dependent on whether [`doCheck = true;`](#var-stdenv-doCheck).
1312
1313This can also be useful in verifying that the package doesn't assume commonly
1314present packages (e.g. `setuptools`).
1315
1316#### Using pythonRelaxDepsHook {#using-pythonrelaxdepshook}
1317
1318It is common for upstream to specify a range of versions for its package
1319dependencies. This makes sense, since it ensures that the package will be built
1320with a subset of packages that is well tested. However, this commonly causes
1321issues when packaging in Nixpkgs, because the dependencies that this package
1322may need are too new or old for the package to build correctly. We also cannot
1323package multiple versions of the same package since this may cause conflicts
1324in `PYTHONPATH`.
1325
1326One way to side step this issue is to relax the dependencies. This can be done
1327by either removing the package version range or by removing the package
1328declaration entirely. This can be done using the `pythonRelaxDepsHook` hook. For
1329example, given the following `requirements.txt` file:
1330
1331```
1332pkg1<1.0
1333pkg2
1334pkg3>=1.0,<=2.0
1335```
1336
1337we can do:
1338
1339```
1340 nativeBuildInputs = [
1341 pythonRelaxDepsHook
1342 ];
1343 pythonRelaxDeps = [
1344 "pkg1"
1345 "pkg3"
1346 ];
1347 pythonRemoveDeps = [
1348 "pkg2"
1349 ];
1350```
1351
1352which would result in the following `requirements.txt` file:
1353
1354```
1355pkg1
1356pkg3
1357```
1358
1359Another option is to pass `true`, that will relax/remove all dependencies, for
1360example:
1361
1362```
1363 nativeBuildInputs = [ pythonRelaxDepsHook ];
1364 pythonRelaxDeps = true;
1365```
1366
1367which would result in the following `requirements.txt` file:
1368
1369```
1370pkg1
1371pkg2
1372pkg3
1373```
1374
1375In general you should always use `pythonRelaxDeps`, because `pythonRemoveDeps`
1376will convert build errors into runtime errors. However `pythonRemoveDeps` may
1377still be useful in exceptional cases, and also to remove dependencies wrongly
1378declared by upstream (for example, declaring `black` as a runtime dependency
1379instead of a dev dependency).
1380
1381Keep in mind that while the examples above are done with `requirements.txt`,
1382`pythonRelaxDepsHook` works by modifying the resulting wheel file, so it should
1383work with any of the [existing hooks](#setup-hooks).
1384
1385#### Using unittestCheckHook {#using-unittestcheckhook}
1386
1387`unittestCheckHook` is a hook which will substitute the setuptools `test` command for a [`checkPhase`](#ssec-check-phase) which runs `python -m unittest discover`:
1388
1389```
1390 nativeCheckInputs = [
1391 unittestCheckHook
1392 ];
1393
1394 unittestFlagsArray = [
1395 "-s" "tests" "-v"
1396 ];
1397```
1398
1399#### Using sphinxHook {#using-sphinxhook}
1400
1401The `sphinxHook` is a helpful tool to build documentation and manpages
1402using the popular Sphinx documentation generator.
1403It is setup to automatically find common documentation source paths and
1404render them using the default `html` style.
1405
1406```
1407 outputs = [
1408 "out"
1409 "doc"
1410 ];
1411
1412 nativeBuildInputs = [
1413 sphinxHook
1414 ];
1415```
1416
1417The hook will automatically build and install the artifact into the
1418`doc` output, if it exists. It also provides an automatic diversion
1419for the artifacts of the `man` builder into the `man` target.
1420
1421```
1422 outputs = [
1423 "out"
1424 "doc"
1425 "man"
1426 ];
1427
1428 # Use multiple builders
1429 sphinxBuilders = [
1430 "singlehtml"
1431 "man"
1432 ];
1433```
1434
1435Overwrite `sphinxRoot` when the hook is unable to find your
1436documentation source root.
1437
1438```
1439 # Configure sphinxRoot for uncommon paths
1440 sphinxRoot = "weird/docs/path";
1441```
1442
1443The hook is also available to packages outside the python ecosystem by
1444referencing it using `sphinxHook` from top-level.
1445
1446### Develop local package {#develop-local-package}
1447
1448As a Python developer you're likely aware of [development mode](http://setuptools.readthedocs.io/en/latest/setuptools.html#development-mode)
1449(`python setup.py develop`); instead of installing the package this command
1450creates a special link to the project code. That way, you can run updated code
1451without having to reinstall after each and every change you make. Development
1452mode is also available. Let's see how you can use it.
1453
1454In the previous Nix expression the source was fetched from a url. We can also
1455refer to a local source instead using `src = ./path/to/source/tree;`
1456
1457If we create a `shell.nix` file which calls [`buildPythonPackage`](#buildpythonpackage-function), and if `src`
1458is a local source, and if the local source has a `setup.py`, then development
1459mode is activated.
1460
1461In the following example, we create a simple environment that has a Python 3.11
1462version of our package in it, as well as its dependencies and other packages we
1463like to have in the environment, all specified with [`propagatedBuildInputs`](#var-stdenv-propagatedBuildInputs).
1464Indeed, we can just add any package we like to have in our environment to
1465[`propagatedBuildInputs`](#var-stdenv-propagatedBuildInputs).
1466
1467```nix
1468with import <nixpkgs> {};
1469with python311Packages;
1470
1471buildPythonPackage rec {
1472 name = "mypackage";
1473 src = ./path/to/package/source;
1474 propagatedBuildInputs = [
1475 pytest
1476 numpy
1477 pkgs.libsndfile
1478 ];
1479}
1480```
1481
1482It is important to note that due to how development mode is implemented on Nix
1483it is not possible to have multiple packages simultaneously in development mode.
1484
1485### Organising your packages {#organising-your-packages}
1486
1487So far we discussed how you can use Python on Nix, and how you can develop with
1488it. We've looked at how you write expressions to package Python packages, and we
1489looked at how you can create environments in which specified packages are
1490available.
1491
1492At some point you'll likely have multiple packages which you would
1493like to be able to use in different projects. In order to minimise unnecessary
1494duplication we now look at how you can maintain a repository with your
1495own packages. The important functions here are `import` and `callPackage`.
1496
1497### Including a derivation using `callPackage` {#including-a-derivation-using-callpackage}
1498
1499Earlier we created a Python environment using [`withPackages`](#python.withpackages-function), and included the
1500`toolz` package via a `let` expression.
1501Let's split the package definition from the environment definition.
1502
1503We first create a function that builds `toolz` in `~/path/to/toolz/release.nix`
1504
1505```nix
1506{ lib
1507, buildPythonPackage
1508, fetchPypi
1509, setuptools
1510, wheel
1511}:
1512
1513buildPythonPackage rec {
1514 pname = "toolz";
1515 version = "0.10.0";
1516 pyproject = true;
1517
1518 src = fetchPypi {
1519 inherit pname version;
1520 hash = "sha256-CP3V73yWSArRHBLUct4hrNMjWZlvaaUlkpm1QP66RWA=";
1521 };
1522
1523 nativeBuildInputs = [
1524 setuptools
1525 wheel
1526 ];
1527
1528 meta = with lib; {
1529 changelog = "https://github.com/pytoolz/toolz/releases/tag/${version}";
1530 homepage = "https://github.com/pytoolz/toolz/";
1531 description = "List processing tools and functional utilities";
1532 license = licenses.bsd3;
1533 maintainers = with maintainers; [ fridh ];
1534 };
1535}
1536```
1537
1538It takes an argument [`buildPythonPackage`](#buildpythonpackage-function). We now call this function using
1539`callPackage` in the definition of our environment
1540
1541```nix
1542with import <nixpkgs> {};
1543
1544( let
1545 toolz = callPackage /path/to/toolz/release.nix {
1546 buildPythonPackage = python310
1547Packages.buildPythonPackage;
1548 };
1549 in python310.withPackages (ps: [
1550 ps.numpy
1551 toolz
1552 ])
1553).env
1554```
1555
1556Important to remember is that the Python version for which the package is made
1557depends on the `python` derivation that is passed to [`buildPythonPackage`](#buildpythonpackage-function). Nix
1558tries to automatically pass arguments when possible, which is why generally you
1559don't explicitly define which `python` derivation should be used. In the above
1560example we use [`buildPythonPackage`](#buildpythonpackage-function) that is part of the set `python3Packages`,
1561and in this case the `python3` interpreter is automatically used.
1562
1563## FAQ {#faq}
1564
1565### How to solve circular dependencies? {#how-to-solve-circular-dependencies}
1566
1567Consider the packages `A` and `B` that depend on each other. When packaging `B`,
1568a solution is to override package `A` not to depend on `B` as an input. The same
1569should also be done when packaging `A`.
1570
1571### How to override a Python package? {#how-to-override-a-python-package}
1572
1573We can override the interpreter and pass `packageOverrides`. In the following
1574example we rename the `pandas` package and build it.
1575
1576```nix
1577with import <nixpkgs> {};
1578
1579(let
1580 python = let
1581 packageOverrides = self: super: {
1582 pandas = super.pandas.overridePythonAttrs(old: {name="foo";});
1583 };
1584 in pkgs.python310.override {
1585 inherit packageOverrides;
1586 };
1587
1588in python.withPackages (ps: [
1589 ps.pandas
1590])).env
1591```
1592
1593Using `nix-build` on this expression will build an environment that contains the
1594package `pandas` but with the new name `foo`.
1595
1596All packages in the package set will use the renamed package. A typical use case
1597is to switch to another version of a certain package. For example, in the
1598Nixpkgs repository we have multiple versions of `django` and `scipy`. In the
1599following example we use a different version of `scipy` and create an
1600environment that uses it. All packages in the Python package set will now use
1601the updated `scipy` version.
1602
1603```nix
1604with import <nixpkgs> {};
1605
1606( let
1607 packageOverrides = self: super: {
1608 scipy = super.scipy_0_17;
1609 };
1610 in (pkgs.python310.override {
1611 inherit packageOverrides;
1612 }).withPackages (ps: [
1613 ps.blaze
1614 ])
1615).env
1616```
1617
1618The requested package `blaze` depends on `pandas` which itself depends on `scipy`.
1619
1620If you want the whole of Nixpkgs to use your modifications, then you can use
1621`overlays` as explained in this manual. In the following example we build a
1622`inkscape` using a different version of `numpy`.
1623
1624```nix
1625let
1626 pkgs = import <nixpkgs> {};
1627 newpkgs = import pkgs.path { overlays = [ (self: super: {
1628 python310 = let
1629 packageOverrides = python-self: python-super: {
1630 numpy = python-super.numpy_1_18;
1631 };
1632 in super.python310.override {inherit packageOverrides;};
1633 } ) ]; };
1634in newpkgs.inkscape
1635```
1636
1637### `python setup.py bdist_wheel` cannot create .whl {#python-setup.py-bdist_wheel-cannot-create-.whl}
1638
1639Executing `python setup.py bdist_wheel` in a `nix-shell`fails with
1640
1641```
1642ValueError: ZIP does not support timestamps before 1980
1643```
1644
1645This is because files from the Nix store (which have a timestamp of the UNIX
1646epoch of January 1, 1970) are included in the .ZIP, but .ZIP archives follow the
1647DOS convention of counting timestamps from 1980.
1648
1649The command `bdist_wheel` reads the `SOURCE_DATE_EPOCH` environment variable,
1650which `nix-shell` sets to 1. Unsetting this variable or giving it a value
1651corresponding to 1980 or later enables building wheels.
1652
1653Use 1980 as timestamp:
1654
1655```shell
1656nix-shell --run "SOURCE_DATE_EPOCH=315532800 python3 setup.py bdist_wheel"
1657```
1658
1659or the current time:
1660
1661```shell
1662nix-shell --run "SOURCE_DATE_EPOCH=$(date +%s) python3 setup.py bdist_wheel"
1663```
1664
1665or unset `SOURCE_DATE_EPOCH`:
1666
1667```shell
1668nix-shell --run "unset SOURCE_DATE_EPOCH; python3 setup.py bdist_wheel"
1669```
1670
1671### `install_data` / `data_files` problems {#install_data-data_files-problems}
1672
1673If you get the following error:
1674
1675```
1676could not create '/nix/store/6l1bvljpy8gazlsw2aw9skwwp4pmvyxw-python-2.7.8/etc':
1677Permission denied
1678```
1679
1680This is a [known bug](https://github.com/pypa/setuptools/issues/130) in
1681`setuptools`. Setuptools `install_data` does not respect `--prefix`. An example
1682of such package using the feature is `pkgs/tools/X11/xpra/default.nix`.
1683
1684As workaround install it as an extra `preInstall` step:
1685
1686```shell
1687${python.pythonOnBuildForHost.interpreter} setup.py install_data --install-dir=$out --root=$out
1688sed -i '/ = data\_files/d' setup.py
1689```
1690
1691### Rationale of non-existent global site-packages {#rationale-of-non-existent-global-site-packages}
1692
1693On most operating systems a global `site-packages` is maintained. This however
1694becomes problematic if you want to run multiple Python versions or have multiple
1695versions of certain libraries for your projects. Generally, you would solve such
1696issues by creating virtual environments using `virtualenv`.
1697
1698On Nix each package has an isolated dependency tree which, in the case of
1699Python, guarantees the right versions of the interpreter and libraries or
1700packages are available. There is therefore no need to maintain a global `site-packages`.
1701
1702If you want to create a Python environment for development, then the recommended
1703method is to use `nix-shell`, either with or without the [`python.buildEnv`](#python.buildenv-function)
1704function.
1705
1706### How to consume Python modules using pip in a virtual environment like I am used to on other Operating Systems? {#how-to-consume-python-modules-using-pip-in-a-virtual-environment-like-i-am-used-to-on-other-operating-systems}
1707
1708While this approach is not very idiomatic from Nix perspective, it can still be
1709useful when dealing with pre-existing projects or in situations where it's not
1710feasible or desired to write derivations for all required dependencies.
1711
1712This is an example of a `default.nix` for a `nix-shell`, which allows to consume
1713a virtual environment created by `venv`, and install Python modules through
1714`pip` the traditional way.
1715
1716Create this `default.nix` file, together with a `requirements.txt` and
1717execute `nix-shell`.
1718
1719```nix
1720with import <nixpkgs> { };
1721
1722let
1723 pythonPackages = python3Packages;
1724in pkgs.mkShell rec {
1725 name = "impurePythonEnv";
1726 venvDir = "./.venv";
1727 buildInputs = [
1728 # A Python interpreter including the 'venv' module is required to bootstrap
1729 # the environment.
1730 pythonPackages.python
1731
1732 # This executes some shell code to initialize a venv in $venvDir before
1733 # dropping into the shell
1734 pythonPackages.venvShellHook
1735
1736 # Those are dependencies that we would like to use from nixpkgs, which will
1737 # add them to PYTHONPATH and thus make them accessible from within the venv.
1738 pythonPackages.numpy
1739 pythonPackages.requests
1740
1741 # In this particular example, in order to compile any binary extensions they may
1742 # require, the Python modules listed in the hypothetical requirements.txt need
1743 # the following packages to be installed locally:
1744 taglib
1745 openssl
1746 git
1747 libxml2
1748 libxslt
1749 libzip
1750 zlib
1751 ];
1752
1753 # Run this command, only after creating the virtual environment
1754 postVenvCreation = ''
1755 unset SOURCE_DATE_EPOCH
1756 pip install -r requirements.txt
1757 '';
1758
1759 # Now we can execute any commands within the virtual environment.
1760 # This is optional and can be left out to run pip manually.
1761 postShellHook = ''
1762 # allow pip to install wheels
1763 unset SOURCE_DATE_EPOCH
1764 '';
1765
1766}
1767```
1768
1769In case the supplied venvShellHook is insufficient, or when Python 2 support is
1770needed, you can define your own shell hook and adapt to your needs like in the
1771following example:
1772
1773```nix
1774with import <nixpkgs> { };
1775
1776let
1777 venvDir = "./.venv";
1778 pythonPackages = python3Packages;
1779in pkgs.mkShell rec {
1780 name = "impurePythonEnv";
1781 buildInputs = [
1782 pythonPackages.python
1783 # Needed when using python 2.7
1784 # pythonPackages.virtualenv
1785 # ...
1786 ];
1787
1788 # This is very close to how venvShellHook is implemented, but
1789 # adapted to use 'virtualenv'
1790 shellHook = ''
1791 SOURCE_DATE_EPOCH=$(date +%s)
1792
1793 if [ -d "${venvDir}" ]; then
1794 echo "Skipping venv creation, '${venvDir}' already exists"
1795 else
1796 echo "Creating new venv environment in path: '${venvDir}'"
1797 # Note that the module venv was only introduced in python 3, so for 2.7
1798 # this needs to be replaced with a call to virtualenv
1799 ${pythonPackages.python.interpreter} -m venv "${venvDir}"
1800 fi
1801
1802 # Under some circumstances it might be necessary to add your virtual
1803 # environment to PYTHONPATH, which you can do here too;
1804 # PYTHONPATH=$PWD/${venvDir}/${pythonPackages.python.sitePackages}/:$PYTHONPATH
1805
1806 source "${venvDir}/bin/activate"
1807
1808 # As in the previous example, this is optional.
1809 pip install -r requirements.txt
1810 '';
1811}
1812```
1813
1814Note that the `pip install` is an imperative action. So every time `nix-shell`
1815is executed it will attempt to download the Python modules listed in
1816requirements.txt. However these will be cached locally within the `virtualenv`
1817folder and not downloaded again.
1818
1819### How to override a Python package from `configuration.nix`? {#how-to-override-a-python-package-from-configuration.nix}
1820
1821If you need to change a package's attribute(s) from `configuration.nix` you could do:
1822
1823```nix
1824 nixpkgs.config.packageOverrides = super: {
1825 python3 = super.python3.override {
1826 packageOverrides = python-self: python-super: {
1827 twisted = python-super.twisted.overridePythonAttrs (oldAttrs: {
1828 src = super.fetchPypi {
1829 pname = "Twisted";
1830 version = "19.10.0";
1831 hash = "sha256-c5S6fycq5yKnTz2Wnc9Zm8TvCTvDkgOHSKSQ8XJKUV0=";
1832 extension = "tar.bz2";
1833 };
1834 });
1835 };
1836 };
1837 };
1838```
1839
1840`python3Packages.twisted` is now globally overridden.
1841All packages and also all NixOS services that reference `twisted`
1842(such as `services.buildbot-worker`) now use the new definition.
1843Note that `python-super` refers to the old package set and `python-self`
1844to the new, overridden version.
1845
1846To modify only a Python package set instead of a whole Python derivation, use
1847this snippet:
1848
1849```nix
1850 myPythonPackages = python3Packages.override {
1851 overrides = self: super: {
1852 twisted = ...;
1853 };
1854 }
1855```
1856
1857### How to override a Python package using overlays? {#how-to-override-a-python-package-using-overlays}
1858
1859Use the following overlay template:
1860
1861```nix
1862self: super: {
1863 python = super.python.override {
1864 packageOverrides = python-self: python-super: {
1865 twisted = python-super.twisted.overrideAttrs (oldAttrs: {
1866 src = super.fetchPypi {
1867 pname = "Twisted";
1868 version = "19.10.0";
1869 hash = "sha256-c5S6fycq5yKnTz2Wnc9Zm8TvCTvDkgOHSKSQ8XJKUV0=";
1870 extension = "tar.bz2";
1871 };
1872 });
1873 };
1874 };
1875}
1876```
1877
1878### How to override a Python package for all Python versions using extensions? {#how-to-override-a-python-package-for-all-python-versions-using-extensions}
1879
1880The following overlay overrides the call to [`buildPythonPackage`](#buildpythonpackage-function) for the
1881`foo` package for all interpreters by appending a Python extension to the
1882`pythonPackagesExtensions` list of extensions.
1883
1884```nix
1885final: prev: {
1886 pythonPackagesExtensions = prev.pythonPackagesExtensions ++ [
1887 (
1888 python-final: python-prev: {
1889 foo = python-prev.foo.overridePythonAttrs (oldAttrs: {
1890 ...
1891 });
1892 }
1893 )
1894 ];
1895}
1896```
1897
1898### How to use Intel’s MKL with numpy and scipy? {#how-to-use-intels-mkl-with-numpy-and-scipy}
1899
1900MKL can be configured using an overlay. See the section "[Using overlays to
1901configure alternatives](#sec-overlays-alternatives-blas-lapack)".
1902
1903### What inputs do `setup_requires`, `install_requires` and `tests_require` map to? {#what-inputs-do-setup_requires-install_requires-and-tests_require-map-to}
1904
1905In a `setup.py` or `setup.cfg` it is common to declare dependencies:
1906
1907* `setup_requires` corresponds to [`nativeBuildInputs`](#var-stdenv-nativeBuildInputs)
1908* `install_requires` corresponds to [`propagatedBuildInputs`](#var-stdenv-propagatedBuildInputs)
1909* `tests_require` corresponds to [`nativeCheckInputs`](#var-stdenv-nativeCheckInputs)
1910
1911### How to enable interpreter optimizations? {#optimizations}
1912
1913The Python interpreters are by default not built with optimizations enabled, because
1914the builds are in that case not reproducible. To enable optimizations, override the
1915interpreter of interest, e.g using
1916
1917```
1918let
1919 pkgs = import ./. {};
1920 mypython = pkgs.python3.override {
1921 enableOptimizations = true;
1922 reproducibleBuild = false;
1923 self = mypython;
1924 };
1925in mypython
1926```
1927
1928### How to add optional dependencies? {#python-optional-dependencies}
1929
1930Some packages define optional dependencies for additional features. With
1931`setuptools` this is called `extras_require` and `flit` calls it
1932`extras-require`, while PEP 621 calls these `optional-dependencies`. A
1933method for supporting this is by declaring the extras of a package in its
1934`passthru`, e.g. in case of the package `dask`
1935
1936```nix
1937passthru.optional-dependencies = {
1938 complete = [ distributed ];
1939};
1940```
1941
1942and letting the package requiring the extra add the list to its dependencies
1943
1944```nix
1945propagatedBuildInputs = [
1946 ...
1947] ++ dask.optional-dependencies.complete;
1948```
1949
1950Note this method is preferred over adding parameters to builders, as that can
1951result in packages depending on different variants and thereby causing
1952collisions.
1953
1954### How to contribute a Python package to nixpkgs? {#tools}
1955
1956Packages inside nixpkgs must use the [`buildPythonPackage`](#buildpythonpackage-function) or [`buildPythonApplication`](#buildpythonapplication-function) function directly,
1957because we can only provide security support for non-vendored dependencies.
1958
1959We recommend [nix-init](https://github.com/nix-community/nix-init) for creating new python packages within nixpkgs,
1960as it already prefetches the source, parses dependencies for common formats and prefills most things in `meta`.
1961
1962### Are Python interpreters built deterministically? {#deterministic-builds}
1963
1964The Python interpreters are now built deterministically. Minor modifications had
1965to be made to the interpreters in order to generate deterministic bytecode. This
1966has security implications and is relevant for those using Python in a
1967`nix-shell`.
1968
1969When the environment variable `DETERMINISTIC_BUILD` is set, all bytecode will
1970have timestamp 1. The [`buildPythonPackage`](#buildpythonpackage-function) function sets `DETERMINISTIC_BUILD=1`
1971and [PYTHONHASHSEED=0](https://docs.python.org/3.11/using/cmdline.html#envvar-PYTHONHASHSEED).
1972Both are also exported in `nix-shell`.
1973
1974### How to provide automatic tests to Python packages? {#automatic-tests}
1975
1976It is recommended to test packages as part of the build process.
1977Source distributions (`sdist`) often include test files, but not always.
1978
1979By default the command `python setup.py test` is run as part of the
1980[`checkPhase`](#ssec-check-phase), but often it is necessary to pass a custom [`checkPhase`](#ssec-check-phase). An
1981example of such a situation is when `py.test` is used.
1982
1983#### Common issues {#common-issues}
1984
1985* Non-working tests can often be deselected. By default [`buildPythonPackage`](#buildpythonpackage-function)
1986 runs `python setup.py test`. which is deprecated. Most Python modules however
1987 do follow the standard test protocol where the pytest runner can be used
1988 instead. `pytest` supports the `-k` and `--ignore` parameters to ignore test
1989 methods or classes as well as whole files. For `pytestCheckHook` these are
1990 conveniently exposed as `disabledTests` and `disabledTestPaths` respectively.
1991
1992 ```nix
1993 buildPythonPackage {
1994 # ...
1995 nativeCheckInputs = [
1996 pytestCheckHook
1997 ];
1998
1999 disabledTests = [
2000 "function_name"
2001 "other_function"
2002 ];
2003
2004 disabledTestPaths = [
2005 "this/file.py"
2006 ];
2007 }
2008 ```
2009
2010* Tests that attempt to access `$HOME` can be fixed by using the following
2011 work-around before running tests (e.g. `preCheck`): `export HOME=$(mktemp -d)`
2012
2013## Contributing {#contributing}
2014
2015### Contributing guidelines {#contributing-guidelines}
2016
2017The following rules are desired to be respected:
2018
2019* Python libraries are called from `python-packages.nix` and packaged with
2020 [`buildPythonPackage`](#buildpythonpackage-function). The expression of a library should be in
2021 `pkgs/development/python-modules/<name>/default.nix`.
2022* Python applications live outside of `python-packages.nix` and are packaged
2023 with [`buildPythonApplication`](#buildpythonapplication-function).
2024* Make sure libraries build for all Python interpreters.
2025* By default we enable tests. Make sure the tests are found and, in the case of
2026 libraries, are passing for all interpreters. If certain tests fail they can be
2027 disabled individually. Try to avoid disabling the tests altogether. In any
2028 case, when you disable tests, leave a comment explaining why.
2029* Commit names of Python libraries should reflect that they are Python
2030 libraries, so write for example `python311Packages.numpy: 1.11 -> 1.12`.
2031 It is highly recommended to specify the current default version to enable
2032 automatic build by ofborg.
2033* Attribute names in `python-packages.nix` as well as `pname`s should match the
2034 library's name on PyPI, but be normalized according to [PEP
2035 0503](https://www.python.org/dev/peps/pep-0503/#normalized-names). This means
2036 that characters should be converted to lowercase and `.` and `_` should be
2037 replaced by a single `-` (foo-bar-baz instead of Foo__Bar.baz).
2038 If necessary, `pname` has to be given a different value within `fetchPypi`.
2039* Packages from sources such as GitHub and GitLab that do not exist on PyPI
2040 should not use a name that is already used on PyPI. When possible, they should
2041 use the package repository name prefixed with the owner (e.g. organization) name
2042 and using a `-` as delimiter.
2043* Attribute names in `python-packages.nix` should be sorted alphanumerically to
2044 avoid merge conflicts and ease locating attributes.
2045
2046## Package set maintenance {#python-package-set-maintenance}
2047
2048The whole Python package set has a lot of packages that do not see regular
2049updates, because they either are a very fragile component in the Python
2050ecosystem, like for example the `hypothesis` package, or packages that have
2051no maintainer, so maintenance falls back to the package set maintainers.
2052
2053### Updating packages in bulk {#python-package-bulk-updates}
2054
2055There is a tool to update alot of python libraries in bulk, it exists at
2056`maintainers/scripts/update-python-libraries` with this repository.
2057
2058It can quickly update minor or major versions for all packages selected
2059and create update commits, and supports the `fetchPypi`, `fetchurl` and
2060`fetchFromGitHub` fetchers. When updating lots of packages that are
2061hosted on GitHub, exporting a `GITHUB_API_TOKEN` is highly recommended.
2062
2063Updating packages in bulk leads to lots of breakages, which is why a
2064stabilization period on the `python-unstable` branch is required.
2065
2066If a package is fragile and often breaks during these bulks updates, it
2067may be reasonable to set `passthru.skipBulkUpdate = true` in the
2068derivation. This decision should not be made on a whim and should
2069always be supported by a qualifying comment.
2070
2071Once the branch is sufficiently stable it should normally be merged
2072into the `staging` branch.
2073
2074An exemplary call to update all python libraries between minor versions
2075would be:
2076
2077```ShellSession
2078$ maintainers/scripts/update-python-libraries --target minor --commit --use-pkgs-prefix pkgs/development/python-modules/**/default.nix
2079```
2080
2081## CPython Update Schedule {#python-cpython-update-schedule}
2082
2083With [PEP 602](https://www.python.org/dev/peps/pep-0602/), CPython now
2084follows a yearly release cadence. In nixpkgs, all supported interpreters
2085are made available, but only the most recent two
2086interpreters package sets are built; this is a compromise between being
2087the latest interpreter, and what the majority of the Python packages support.
2088
2089New CPython interpreters are released in October. Generally, it takes some
2090time for the majority of active Python projects to support the latest stable
2091interpreter. To help ease the migration for Nixpkgs users
2092between Python interpreters the schedule below will be used:
2093
2094| When | Event |
2095| --- | --- |
2096| After YY.11 Release | Bump CPython package set window. The latest and previous latest stable should now be built. |
2097| After YY.05 Release | Bump default CPython interpreter to latest stable. |
2098
2099In practice, this means that the Python community will have had a stable interpreter
2100for ~2 months before attempting to update the package set. And this will
2101allow for ~7 months for Python applications to support the latest interpreter.