pyrox.dev
1# Python {#python}
2
3## Reference {#reference}
4
5### Interpreters {#interpreters}
6
7@python-interpreter-table@
8
9The Nix expressions for the interpreters can be found in
10`pkgs/development/interpreters/python`.
11
12All packages depending on any Python interpreter get appended
13`out/{python.sitePackages}` to `$PYTHONPATH` if such directory
14exists.
15
16#### Missing `tkinter` module standard library {#missing-tkinter-module-standard-library}
17
18To reduce closure size the `Tkinter`/`tkinter` is available as a separate package, `pythonPackages.tkinter`.
19
20#### Attributes on interpreters packages {#attributes-on-interpreters-packages}
21
22Each interpreter has the following attributes:
23
24- `libPrefix`. Name of the folder in `${python}/lib/` for corresponding interpreter.
25- `interpreter`. Alias for `${python}/bin/${executable}`.
26- `buildEnv`. Function to build python interpreter environments with extra packages bundled together. See [](#python.buildenv-function) for usage and documentation.
27- `withPackages`. Simpler interface to `buildEnv`. See [](#python.withpackages-function) for usage and documentation.
28- `sitePackages`. Alias for `lib/${libPrefix}/site-packages`.
29- `executable`. Name of the interpreter executable, e.g. `python3.10`.
30- `pkgs`. Set of Python packages for that specific interpreter. The package set can be modified by overriding the interpreter and passing `packageOverrides`.
31
32### Building packages and applications {#building-packages-and-applications}
33
34Python libraries and applications that use tools to follow PEP 517 (e.g. `setuptools` or `hatchling`, etc.) or
35previous tools such as `distutils` are typically built with respectively the [`buildPythonPackage`](#buildpythonpackage-function) and
36[`buildPythonApplication`](#buildpythonapplication-function) functions. These two functions also support installing a `wheel`.
37
38All Python packages reside in `pkgs/top-level/python-packages.nix` and all
39applications elsewhere. In case a package is used as both a library and an
40application, then the package should be in `pkgs/top-level/python-packages.nix`
41since only those packages are made available for all interpreter versions. The
42preferred location for library expressions is in
43`pkgs/development/python-modules`. It is important that these packages are
44called from `pkgs/top-level/python-packages.nix` and not elsewhere, to guarantee
45the right version of the package is built.
46
47Based on the packages defined in `pkgs/top-level/python-packages.nix` an
48attribute set is created for each available Python interpreter. The available
49sets are
50
51* `pkgs.python27Packages`
52* `pkgs.python3Packages`
53* `pkgs.python310Packages`
54* `pkgs.python311Packages`
55* `pkgs.python312Packages`
56* `pkgs.python313Packages`
57* `pkgs.python314Packages`
58* `pkgs.pypy27Packages`
59* `pkgs.pypy310Packages`
60
61and the aliases
62
63* `pkgs.python2Packages` pointing to `pkgs.python27Packages`
64* `pkgs.python3Packages` pointing to `pkgs.python312Packages`
65* `pkgs.pythonPackages` pointing to `pkgs.python2Packages`
66* `pkgs.pypy2Packages` pointing to `pkgs.pypy27Packages`
67* `pkgs.pypy3Packages` pointing to `pkgs.pypy310Packages`
68* `pkgs.pypyPackages` pointing to `pkgs.pypy2Packages`
69
70
71#### `buildPythonPackage` function {#buildpythonpackage-function}
72
73The `buildPythonPackage` function has its name binding in
74`pkgs/development/interpreters/python/python-packages-base.nix` and is
75implemented in `pkgs/development/interpreters/python/mk-python-derivation.nix`
76using setup hooks.
77
78The following is an example:
79
80```nix
81{
82 lib,
83 buildPythonPackage,
84 fetchPypi,
85
86 # build-system
87 setuptools,
88 setuptools-scm,
89
90 # dependencies
91 attrs,
92 pluggy,
93 py,
94 setuptools,
95 six,
96
97 # tests
98 hypothesis,
99}:
100
101buildPythonPackage rec {
102 pname = "pytest";
103 version = "3.3.1";
104 pyproject = true;
105
106 src = fetchPypi {
107 inherit pname version;
108 hash = "sha256-z4Q23FnYaVNG/NOrKW3kZCXsqwDWQJbOvnn7Ueyy65M=";
109 };
110
111 postPatch = ''
112 # don't test bash builtins
113 rm testing/test_argcomplete.py
114 '';
115
116 build-system = [
117 setuptools
118 setuptools-scm
119 ];
120
121 dependencies = [
122 attrs
123 py
124 setuptools
125 six
126 pluggy
127 ];
128
129 nativeCheckInputs = [
130 hypothesis
131 ];
132
133 meta = {
134 changelog = "https://github.com/pytest-dev/pytest/releases/tag/${version}";
135 description = "Framework for writing tests";
136 homepage = "https://github.com/pytest-dev/pytest";
137 license = lib.licenses.mit;
138 maintainers = with lib.maintainers; [
139 domenkozar
140 lovek323
141 madjar
142 lsix
143 ];
144 };
145}
146```
147
148The `buildPythonPackage` mainly does four things:
149
150* In the [`buildPhase`](#build-phase), it calls `${python.pythonOnBuildForHost.interpreter} -m build --wheel` to
151 build a wheel binary zipfile.
152* In the [`installPhase`](#ssec-install-phase), it installs the wheel file using `${python.pythonOnBuildForHost.interpreter} -m installer *.whl`.
153* In the [`postFixup`](#var-stdenv-postFixup) phase, the `wrapPythonPrograms` bash function is called to
154 wrap all programs in the `$out/bin/*` directory to include `$PATH`
155 environment variable and add dependent libraries to script's `sys.path`.
156* In the [`installCheck`](#ssec-installCheck-phase) phase, `${python.interpreter} -m pytest` is run.
157
158By default tests are run because [`doCheck = true`](#var-stdenv-doCheck). Test dependencies, like
159e.g. the test runner, should be added to [`nativeCheckInputs`](#var-stdenv-nativeCheckInputs).
160
161By default `meta.platforms` is set to the same value
162as the interpreter unless overridden otherwise.
163
164##### `buildPythonPackage` parameters {#buildpythonpackage-parameters}
165
166All parameters from [`stdenv.mkDerivation`](#sec-using-stdenv) function are still supported. The
167following are specific to `buildPythonPackage`:
168
169* `catchConflicts ? true`: If `true`, abort package build if a package name
170 appears more than once in dependency tree. Default is `true`.
171* `disabled ? false`: If `true`, package is not built for the particular Python
172 interpreter version.
173* `dontWrapPythonPrograms ? false`: Skip wrapping of Python programs.
174* `permitUserSite ? false`: Skip setting the `PYTHONNOUSERSITE` environment
175 variable in wrapped programs.
176* `pyproject`: Whether the pyproject format should be used. As all other formats
177 are deprecated, you are recommended to set this to `true`. When you do so,
178 `pypaBuildHook` will be used, and you can add the required build dependencies
179 from `build-system.requires` to `build-system`. Note that the pyproject
180 format falls back to using `setuptools`, so you can use `pyproject = true`
181 even if the package only has a `setup.py`. When set to `false`, you can
182 use the existing [hooks](#setup-hooks) or provide your own logic to build the
183 package. This can be useful for packages that don't support the pyproject
184 format. When unset, the legacy `setuptools` hooks are used for backwards
185 compatibility.
186* `makeWrapperArgs ? []`: A list of strings. Arguments to be passed to
187 [`makeWrapper`](#fun-makeWrapper), which wraps generated binaries. By default, the arguments to
188 [`makeWrapper`](#fun-makeWrapper) set `PATH` and `PYTHONPATH` environment variables before calling
189 the binary. Additional arguments here can allow a developer to set environment
190 variables which will be available when the binary is run. For example,
191 `makeWrapperArgs = ["--set" "FOO" "BAR" "--set" "BAZ" "QUX"]`.
192
193 ::: {.note}
194 When `__structuredAttrs = false`, the attribute `makeWrapperArgs` is passed as a space-separated string to the build script. Developers should use `prependToVar` or `appendToVar` to add arguments to it in build phases, or use `__structuredAttrs = true` to ensure that `makeWrapperArgs` is passed as a Bash array.
195
196 For compatibility purposes,
197 when `makeWrapperArgs` shell variable is specified as a space-separated string (instead of a Bash array) in the build script, the string content is Bash-expanded before concatenated into the `wrapProgram` command. Still, developers should not rely on such behaviours, but use `__structuredAttrs = true` to specify flags containing spaces (e.g. `makeWrapperArgs = [ "--set" "GREETING" "Hello, world!" ]`), or use -pre and -post phases to specify flags with Bash-expansions (e.g. `preFixup = ''makeWrapperArgs+=(--prefix PATH : "$SOME_PATH")`'').
198 :::
199
200* `namePrefix`: Prepends text to `${name}` parameter. In case of libraries, this
201 defaults to `"python3.8-"` for Python 3.8, etc., and in case of applications to `""`.
202* `pypaBuildFlags ? []`: A list of strings. Arguments to be passed to `python -m build --wheel`.
203* `pythonPath ? []`: List of packages to be added into `$PYTHONPATH`. Packages
204 in `pythonPath` are not propagated (contrary to [`propagatedBuildInputs`](#var-stdenv-propagatedBuildInputs)).
205* `preShellHook`: Hook to execute commands before `shellHook`.
206* `postShellHook`: Hook to execute commands after `shellHook`.
207* `removeBinByteCode ? true`: Remove bytecode from `/bin`. Bytecode is only
208 created when the filenames end with `.py`.
209* `setupPyGlobalFlags ? []`: List of flags passed to `setup.py` command.
210* `setupPyBuildFlags ? []`: List of flags passed to `setup.py build_ext` command.
211
212The [`stdenv.mkDerivation`](#sec-using-stdenv) function accepts various parameters for describing
213build inputs (see "Specifying dependencies"). The following are of special
214interest for Python packages, either because these are primarily used, or
215because their behaviour is different:
216
217* `nativeBuildInputs ? []`: Build-time only dependencies. Typically executables.
218* `build-system ? []`: Build-time only Python dependencies. Items listed in `build-system.requires`/`setup_requires`.
219* `buildInputs ? []`: Build and/or run-time dependencies that need to be
220 compiled for the host machine. Typically non-Python libraries which are being
221 linked.
222* `nativeCheckInputs ? []`: Dependencies needed for running the [`checkPhase`](#ssec-check-phase). These
223 are added to [`nativeBuildInputs`](#var-stdenv-nativeBuildInputs) when [`doCheck = true`](#var-stdenv-doCheck). Items listed in
224 `tests_require` go here.
225* `dependencies ? []`: Aside from propagating dependencies,
226 `buildPythonPackage` also injects code into and wraps executables with the
227 paths included in this list. Items listed in `install_requires` go here.
228* `optional-dependencies ? { }`: Optional feature flagged dependencies. Items listed in `extras_require` go here.
229
230
231##### Overriding Python packages {#overriding-python-packages}
232
233The `buildPythonPackage` function has a `overridePythonAttrs` method that can be
234used to override the package. In the following example we create an environment
235where we have the `blaze` package using an older version of `pandas`. We
236override first the Python interpreter and pass `packageOverrides` which contains
237the overrides for packages in the package set.
238
239```nix
240with import <nixpkgs> { };
241
242(
243 let
244 python =
245 let
246 packageOverrides = self: super: {
247 pandas = super.pandas.overridePythonAttrs (old: rec {
248 version = "0.19.1";
249 src = fetchPypi {
250 pname = "pandas";
251 inherit version;
252 hash = "sha256-JQn+rtpy/OA2deLszSKEuxyttqBzcAil50H+JDHUdCE=";
253 };
254 });
255 };
256 in
257 pkgs.python3.override {
258 inherit packageOverrides;
259 self = python;
260 };
261
262 in
263 python.withPackages (ps: [ ps.blaze ])
264).env
265```
266
267The next example shows a non trivial overriding of the `blas` implementation to
268be used through out all of the Python package set:
269
270```nix
271{
272 python3MyBlas = pkgs.python3.override {
273 packageOverrides = self: super: {
274 # We need toPythonModule for the package set to evaluate this
275 blas = super.toPythonModule (
276 super.pkgs.blas.override {
277 blasProvider = super.pkgs.mkl;
278 }
279 );
280 lapack = super.toPythonModule (
281 super.pkgs.lapack.override {
282 lapackProvider = super.pkgs.mkl;
283 }
284 );
285 };
286 };
287}
288```
289
290This is particularly useful for numpy and scipy users who want to gain speed with other blas implementations.
291Note that using `scipy = super.scipy.override { blas = super.pkgs.mkl; };` will likely result in
292compilation issues, because scipy dependencies need to use the same blas implementation as well.
293
294#### `buildPythonApplication` function {#buildpythonapplication-function}
295
296The [`buildPythonApplication`](#buildpythonapplication-function) function is practically the same as
297[`buildPythonPackage`](#buildpythonpackage-function). The main purpose of this function is to build a Python
298package where one is interested only in the executables, and not importable
299modules. For that reason, when adding this package to a [`python.buildEnv`](#python.buildenv-function), the
300modules won't be made available.
301
302Another difference is that [`buildPythonPackage`](#buildpythonpackage-function) by default prefixes the names of
303the packages with the version of the interpreter. Because this is irrelevant for
304applications, the prefix is omitted.
305
306When packaging a Python application with [`buildPythonApplication`](#buildpythonapplication-function), it should be
307called with `callPackage` and passed `python3` or `python3Packages` (possibly
308specifying an interpreter version), like this:
309
310```nix
311{
312 lib,
313 python3Packages,
314 fetchPypi,
315}:
316
317python3Packages.buildPythonApplication rec {
318 pname = "luigi";
319 version = "2.7.9";
320 pyproject = true;
321
322 src = fetchPypi {
323 inherit pname version;
324 hash = "sha256-Pe229rT0aHwA98s+nTHQMEFKZPo/yw6sot8MivFDvAw=";
325 };
326
327 build-system = with python3Packages; [
328 setuptools
329 ];
330
331 dependencies = with python3Packages; [
332 tornado
333 python-daemon
334 ];
335
336 meta = {
337 # ...
338 };
339}
340```
341
342This is then added to `pkgs/by-name` just as any other application would be.
343
344Since the package is an application, a consumer doesn't need to care about
345Python versions or modules, which is why they don't go in `python3Packages`.
346
347#### `toPythonApplication` function {#topythonapplication-function}
348
349A distinction is made between applications and libraries, however, sometimes a
350package is used as both. In this case the package is added as a library to
351`python-packages.nix` and as an application to `pkgs/by-name`. To reduce
352duplication the `toPythonApplication` can be used to convert a library to an
353application.
354
355The Nix expression shall use [`buildPythonPackage`](#buildpythonpackage-function) and be called from
356`python-packages.nix`. A reference shall be created from `pkgs/by-name` to
357the attribute in `python-packages.nix`, and the `toPythonApplication` shall be
358applied to the reference:
359
360```nix
361{
362 python3Packages,
363}:
364
365python3Packages.toPythonApplication python3Packages.youtube-dl
366```
367
368#### `toPythonModule` function {#topythonmodule-function}
369
370In some cases, such as bindings, a package is created using
371[`stdenv.mkDerivation`](#sec-using-stdenv) and added as attribute in `pkgs/by-name` or in `all-packages.nix`. The Python
372bindings should be made available from `python-packages.nix`. The
373`toPythonModule` function takes a derivation and makes certain Python-specific
374modifications.
375
376```nix
377{
378 opencv = toPythonModule (
379 pkgs.opencv.override {
380 enablePython = true;
381 pythonPackages = self;
382 }
383 );
384}
385```
386
387Do pay attention to passing in the right Python version!
388
389#### `mkPythonMetaPackage` function {#mkpythonmetapackage-function}
390
391This will create a meta package containing [metadata files](https://packaging.python.org/en/latest/specifications/recording-installed-packages/) to satisfy a dependency on a package, without it actually having been installed into the environment.
392In nixpkgs this is used to package Python packages with split binary/source distributions such as [psycopg2](https://pypi.org/project/psycopg2/)/[psycopg2-binary](https://pypi.org/project/psycopg2-binary/).
393
394```nix
395mkPythonMetaPackage {
396 pname = "psycopg2-binary";
397 inherit (psycopg2) optional-dependencies version;
398 dependencies = [ psycopg2 ];
399 meta = {
400 inherit (psycopg2.meta) description homepage;
401 };
402}
403```
404
405#### `mkPythonEditablePackage` function {#mkpythoneditablepackage-function}
406
407When developing Python packages it's common to install packages in [editable mode](https://setuptools.pypa.io/en/latest/userguide/development_mode.html).
408Like `mkPythonMetaPackage` this function exists to create an otherwise empty package, but also containing a pointer to an impure location outside the Nix store that can be changed without rebuilding.
409
410The editable root is passed as a string. Normally `.pth` files contains absolute paths to the mutable location. This isn't always ergonomic with Nix, so environment variables are expanded at runtime.
411This means that a shell hook setting up something like a `$REPO_ROOT` variable can be used as the relative package root.
412
413As an implementation detail, the [PEP-518](https://peps.python.org/pep-0518/) `build-system` specified won't be used, but instead the editable package will be built using [hatchling](https://pypi.org/project/hatchling/).
414The `build-system`'s provided will instead become runtime dependencies of the editable package.
415
416Note that overriding packages deeper in the dependency graph _can_ work, but it's not the primary use case and overriding existing packages can make others break in unexpected ways.
417
418```nix
419{
420 pkgs ? import <nixpkgs> { },
421}:
422
423let
424 pyproject = pkgs.lib.importTOML ./pyproject.toml;
425
426 myPython = pkgs.python.override {
427 self = myPython;
428 packageOverrides = pyfinal: pyprev: {
429 # An editable package with a script that loads our mutable location
430 my-editable = pyfinal.mkPythonEditablePackage {
431 # Inherit project metadata from pyproject.toml
432 pname = pyproject.project.name;
433 inherit (pyproject.project) version;
434
435 # The editable root passed as a string
436 root = "$REPO_ROOT/src"; # Use environment variable expansion at runtime
437
438 # Inject a script (other PEP-621 entrypoints are also accepted)
439 inherit (pyproject.project) scripts;
440 };
441 };
442 };
443
444 pythonEnv = myPython.withPackages (ps: [ ps.my-editable ]);
445
446in
447pkgs.mkShell {
448 packages = [ pythonEnv ];
449}
450```
451
452#### `python.buildEnv` function {#python.buildenv-function}
453
454Python environments can be created using the low-level `pkgs.buildEnv` function.
455This example shows how to create an environment that has the Pyramid Web Framework.
456Saving the following as `default.nix`
457
458```nix
459with import <nixpkgs> { };
460
461python3.buildEnv.override {
462 extraLibs = [ python3Packages.pyramid ];
463 ignoreCollisions = true;
464}
465```
466
467and running `nix-build` will create
468
469```
470/nix/store/cf1xhjwzmdki7fasgr4kz6di72ykicl5-python-2.7.8-env
471```
472
473with wrapped binaries in `bin/`.
474
475You can also use the `env` attribute to create local environments with needed
476packages installed. This is somewhat comparable to `virtualenv`. For example,
477running `nix-shell` with the following `shell.nix`
478
479```nix
480with import <nixpkgs> { };
481
482(python3.buildEnv.override {
483 extraLibs = with python3Packages; [
484 numpy
485 requests
486 ];
487}).env
488```
489
490will drop you into a shell where Python will have the
491specified packages in its path.
492
493##### `python.buildEnv` arguments {#python.buildenv-arguments}
494
495
496* `extraLibs`: List of packages installed inside the environment.
497* `postBuild`: Shell command executed after the build of environment.
498* `ignoreCollisions`: Ignore file collisions inside the environment (default is `false`).
499* `permitUserSite`: Skip setting the `PYTHONNOUSERSITE` environment variable in
500 wrapped binaries in the environment.
501
502#### `python.withPackages` function {#python.withpackages-function}
503
504The [`python.withPackages`](#python.withpackages-function) function provides a simpler interface to the [`python.buildEnv`](#python.buildenv-function) functionality.
505It takes a function as an argument that is passed the set of python packages and returns the list
506of the packages to be included in the environment. Using the [`withPackages`](#python.withpackages-function) function, the previous
507example for the Pyramid Web Framework environment can be written like this:
508
509```nix
510with import <nixpkgs> { };
511
512python.withPackages (ps: [ ps.pyramid ])
513```
514
515[`withPackages`](#python.withpackages-function) passes the correct package set for the specific interpreter
516version as an argument to the function. In the above example, `ps` equals
517`pythonPackages`. But you can also easily switch to using python3:
518
519```nix
520with import <nixpkgs> { };
521
522python3.withPackages (ps: [ ps.pyramid ])
523```
524
525Now, `ps` is set to `python3Packages`, matching the version of the interpreter.
526
527As [`python.withPackages`](#python.withpackages-function) uses [`python.buildEnv`](#python.buildenv-function) under the hood, it also
528supports the `env` attribute. The `shell.nix` file from the previous section can
529thus be also written like this:
530
531```nix
532with import <nixpkgs> { };
533
534(python3.withPackages (
535 ps: with ps; [
536 numpy
537 requests
538 ]
539)).env
540```
541
542In contrast to [`python.buildEnv`](#python.buildenv-function), [`python.withPackages`](#python.withpackages-function) does not support the
543more advanced options such as `ignoreCollisions = true` or `postBuild`. If you
544need them, you have to use [`python.buildEnv`](#python.buildenv-function).
545
546Python 2 namespace packages may provide `__init__.py` that collide. In that case
547[`python.buildEnv`](#python.buildenv-function) should be used with `ignoreCollisions = true`.
548
549#### Setup hooks {#setup-hooks}
550
551The following are setup hooks specifically for Python packages. Most of these
552are used in [`buildPythonPackage`](#buildpythonpackage-function).
553
554- `eggUnpackhook` to move an egg to the correct folder so it can be installed
555 with the `eggInstallHook`
556- `eggBuildHook` to skip building for eggs.
557- `eggInstallHook` to install eggs.
558- `pypaBuildHook` to build a wheel using
559 [`pypa/build`](https://pypa-build.readthedocs.io/en/latest/index.html) and
560 PEP 517/518. Note a build system (e.g. `setuptools` or `flit`) should still
561 be added as `build-system`.
562- `pypaInstallHook` to install wheels.
563- `pytestCheckHook` to run tests with `pytest`. See [example usage](#using-pytestcheckhook).
564- `pythonCatchConflictsHook` to fail if the package depends on two different versions of the same dependency.
565- `pythonImportsCheckHook` to check whether importing the listed modules works.
566- `pythonRelaxDepsHook` will relax Python dependencies restrictions for the package.
567 See [example usage](#using-pythonrelaxdepshook).
568- `pythonRemoveBinBytecode` to remove bytecode from the `/bin` folder.
569- `setuptoolsBuildHook` to build a wheel using `setuptools`.
570- `sphinxHook` to build documentation and manpages using Sphinx.
571- `venvShellHook` to source a Python 3 `venv` at the `venvDir` location. A
572 `venv` is created if it does not yet exist. `postVenvCreation` can be used to
573 to run commands only after venv is first created.
574- `wheelUnpackHook` to move a wheel to the correct folder so it can be installed
575 with the `pipInstallHook`.
576- `unittestCheckHook` will run tests with `python -m unittest discover`. See [example usage](#using-unittestcheckhook).
577
578## User Guide {#user-guide}
579
580### Using Python {#using-python}
581
582#### Overview {#overview}
583
584Several versions of the Python interpreter are available on Nix, as well as a
585high amount of packages. The attribute `python3` refers to the default
586interpreter, which is currently CPython 3.12. The attribute `python` refers to
587CPython 2.7 for backwards-compatibility. It is also possible to refer to
588specific versions, e.g. `python312` refers to CPython 3.12, and `pypy` refers to
589the default PyPy interpreter.
590
591Python is used a lot, and in different ways. This affects also how it is
592packaged. In the case of Python on Nix, an important distinction is made between
593whether the package is considered primarily an application, or whether it should
594be used as a library, i.e., of primary interest are the modules in
595`site-packages` that should be importable.
596
597In the Nixpkgs tree Python applications can be found throughout, depending on
598what they do, and are called from the main package set. Python libraries,
599however, are in separate sets, with one set per interpreter version.
600
601The interpreters have several common attributes. One of these attributes is
602`pkgs`, which is a package set of Python libraries for this specific
603interpreter. E.g., the `toolz` package corresponding to the default interpreter
604is `python3.pkgs.toolz`, and the CPython 3.12 version is `python312.pkgs.toolz`.
605The main package set contains aliases to these package sets, e.g.
606`pythonPackages` refers to `python.pkgs` and `python312Packages` to
607`python312.pkgs`.
608
609#### Installing Python and packages {#installing-python-and-packages}
610
611The Nix and NixOS manuals explain how packages are generally installed. In the
612case of Python and Nix, it is important to make a distinction between whether the
613package is considered an application or a library.
614
615Applications on Nix are typically installed into your user profile imperatively
616using `nix-env -i`, and on NixOS declaratively by adding the package name to
617`environment.systemPackages` in `/etc/nixos/configuration.nix`. Dependencies
618such as libraries are automatically installed and should not be installed
619explicitly.
620
621The same goes for Python applications. Python applications can be installed in
622your profile, and will be wrapped to find their exact library dependencies,
623without impacting other applications or polluting your user environment.
624
625But Python libraries you would like to use for development cannot be installed,
626at least not individually, because they won't be able to find each other
627resulting in import errors. Instead, it is possible to create an environment
628with [`python.buildEnv`](#python.buildenv-function) or [`python.withPackages`](#python.withpackages-function) where the interpreter and other
629executables are wrapped to be able to find each other and all of the modules.
630
631In the following examples we will start by creating a simple, ad-hoc environment
632with a nix-shell that has `numpy` and `toolz` in Python 3.12; then we will create
633a re-usable environment in a single-file Python script; then we will create a
634full Python environment for development with this same environment.
635
636Philosophically, this should be familiar to users who are used to a `venv` style
637of development: individual projects create their own Python environments without
638impacting the global environment or each other.
639
640#### Ad-hoc temporary Python environment with `nix-shell` {#ad-hoc-temporary-python-environment-with-nix-shell}
641
642The simplest way to start playing with the way nix wraps and sets up Python
643environments is with `nix-shell` at the cmdline. These environments create a
644temporary shell session with a Python and a *precise* list of packages (plus
645their runtime dependencies), with no other Python packages in the Python
646interpreter's scope.
647
648To create a Python 3.12 session with `numpy` and `toolz` available, run:
649
650```sh
651$ nix-shell -p 'python312.withPackages(ps: with ps; [ numpy toolz ])'
652```
653
654By default `nix-shell` will start a `bash` session with this interpreter in our
655`PATH`, so if we then run:
656
657```Python console
658[nix-shell:~/src/nixpkgs]$ python3
659Python 3.12.4 (main, Jun 6 2024, 18:26:44) [GCC 13.3.0] on linux
660Type "help", "copyright", "credits" or "license" for more information.
661>>> import numpy; import toolz
662```
663
664Note that no other modules are in scope, even if they were imperatively
665installed into our user environment as a dependency of a Python application:
666
667```Python console
668>>> import requests
669Traceback (most recent call last):
670 File "<stdin>", line 1, in <module>
671ModuleNotFoundError: No module named 'requests'
672```
673
674We can add as many additional modules onto the `nix-shell` as we need, and we
675will still get 1 wrapped Python interpreter. We can start the interpreter
676directly like so:
677
678```sh
679$ nix-shell -p "python312.withPackages (ps: with ps; [ numpy toolz requests ])" --run python3
680Python 3.12.4 (main, Jun 6 2024, 18:26:44) [GCC 13.3.0] on linux
681Type "help", "copyright", "credits" or "license" for more information.
682>>> import requests
683>>>
684```
685
686Notice that this time it built a new Python environment, which now includes
687`requests`. Building an environment just creates wrapper scripts that expose the
688selected dependencies to the interpreter while re-using the actual modules. This
689means if any other env has installed `requests` or `numpy` in a different
690context, we don't need to recompile them -- we just recompile the wrapper script
691that sets up an interpreter pointing to them. This matters much more for "big"
692modules like `pytorch` or `tensorflow`.
693
694Module names usually match their names on [pypi.org](https://pypi.org/), but
695normalized according to PEP 503/508. (e.g. Foo__Bar.baz -> foo-bar-baz)
696You can use the [Nixpkgs search website](https://nixos.org/nixos/packages.html)
697to find them as well (along with non-python packages).
698
699At this point we can create throwaway experimental Python environments with
700arbitrary dependencies. This is a good way to get a feel for how the Python
701interpreter and dependencies work in Nix and NixOS, but to do some actual
702development, we'll want to make it a bit more persistent.
703
704##### Running Python scripts and using `nix-shell` as shebang {#running-python-scripts-and-using-nix-shell-as-shebang}
705
706Sometimes, we have a script whose header looks like this:
707
708```python
709#!/usr/bin/env python3
710import numpy as np
711a = np.array([1,2])
712b = np.array([3,4])
713print(f"The dot product of {a} and {b} is: {np.dot(a, b)}")
714```
715
716Executing this script requires a `python3` that has `numpy`. Using what we learned
717in the previous section, we could startup a shell and just run it like so:
718
719```ShellSession
720$ nix-shell -p 'python312.withPackages (ps: with ps; [ numpy ])' --run 'python3 foo.py'
721The dot product of [1 2] and [3 4] is: 11
722```
723
724But if we maintain the script ourselves, and if there are more dependencies, it
725may be nice to encode those dependencies in source to make the script re-usable
726without that bit of knowledge. That can be done by using `nix-shell` as a
727[shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)), like so:
728
729```python
730#!/usr/bin/env nix-shell
731#!nix-shell -i python3 -p "python3.withPackages(ps: [ ps.numpy ])"
732import numpy as np
733a = np.array([1,2])
734b = np.array([3,4])
735print(f"The dot product of {a} and {b} is: {np.dot(a, b)}")
736```
737
738Then we execute it, without requiring any environment setup at all!
739
740```sh
741$ ./foo.py
742The dot product of [1 2] and [3 4] is: 11
743```
744
745If the dependencies are not available on the host where `foo.py` is executed, it
746will build or download them from a Nix binary cache prior to starting up, prior
747that it is executed on a machine with a multi-user nix installation.
748
749This provides a way to ship a self bootstrapping Python script, akin to a
750statically linked binary, where it can be run on any machine (provided nix is
751installed) without having to assume that `numpy` is installed globally on the
752system.
753
754By default it is pulling the import checkout of Nixpkgs itself from our nix
755channel, which is nice as it cache aligns with our other package builds, but we
756can make it fully reproducible by pinning the `nixpkgs` import:
757
758```python
759#!/usr/bin/env nix-shell
760#!nix-shell -i python3 -p "python3.withPackages (ps: [ ps.numpy ])"
761#!nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/e51209796c4262bfb8908e3d6d72302fe4e96f5f.tar.gz
762import numpy as np
763a = np.array([1,2])
764b = np.array([3,4])
765print(f"The dot product of {a} and {b} is: {np.dot(a, b)}")
766```
767
768This will execute with the exact same versions of Python 3.10, numpy, and system
769dependencies a year from now as it does today, because it will always use
770exactly git commit `e51209796c4262bfb8908e3d6d72302fe4e96f5f` of Nixpkgs for all
771of the package versions.
772
773This is also a great way to ensure the script executes identically on different
774servers.
775
776##### Load environment from `.nix` expression {#load-environment-from-.nix-expression}
777
778We've now seen how to create an ad-hoc temporary shell session, and how to
779create a single script with Python dependencies, but in the course of normal
780development we're usually working in an entire package repository.
781
782As explained [in the `nix-shell` section](https://nixos.org/manual/nix/stable/command-ref/nix-shell) of the Nix manual, `nix-shell` can also load an expression from a `.nix` file.
783Say we want to have Python 3.12, `numpy` and `toolz`, like before,
784in an environment. We can add a `shell.nix` file describing our dependencies:
785
786```nix
787with import <nixpkgs> { };
788(python312.withPackages (
789 ps: with ps; [
790 numpy
791 toolz
792 ]
793)).env
794```
795
796And then at the command line, just typing `nix-shell` produces the same
797environment as before. In a normal project, we'll likely have many more
798dependencies; this can provide a way for developers to share the environments
799with each other and with CI builders.
800
801What's happening here?
802
8031. We begin with importing the Nix Packages collections. `import <nixpkgs>`
804 imports the `<nixpkgs>` function, `{}` calls it and the `with` statement
805 brings all attributes of `nixpkgs` in the local scope. These attributes form
806 the main package set.
8072. Then we create a Python 3.12 environment with the [`withPackages`](#python.withpackages-function) function, as before.
8083. The [`withPackages`](#python.withpackages-function) function expects us to provide a function as an argument
809 that takes the set of all Python packages and returns a list of packages to
810 include in the environment. Here, we select the packages `numpy` and `toolz`
811 from the package set.
812
813To combine this with `mkShell` you can:
814
815```nix
816with import <nixpkgs> { };
817let
818 pythonEnv = python312.withPackages (ps: [
819 ps.numpy
820 ps.toolz
821 ]);
822in
823mkShell {
824 packages = [
825 pythonEnv
826
827 black
828 mypy
829
830 libffi
831 openssl
832 ];
833}
834```
835
836This will create a unified environment that has not just our Python interpreter
837and its Python dependencies, but also tools like `black` or `mypy` and libraries
838like `libffi` the `openssl` in scope. This is generic and can span any number of
839tools or languages across the Nixpkgs ecosystem.
840
841##### Installing environments globally on the system {#installing-environments-globally-on-the-system}
842
843Up to now, we've been creating environments scoped to an ad-hoc shell session,
844or a single script, or a single project. This is generally advisable, as it
845avoids pollution across contexts.
846
847However, sometimes we know we will often want a Python with some basic packages,
848and want this available without having to enter into a shell or build context.
849This can be useful to have things like vim/emacs editors and plugins or shell
850tools "just work" without having to set them up, or when running other software
851that expects packages to be installed globally.
852
853To create your own custom environment, create a file in `~/.config/nixpkgs/overlays/`
854that looks like this:
855
856```nix
857# ~/.config/nixpkgs/overlays/myEnv.nix
858self: super: {
859 myEnv = super.buildEnv {
860 name = "myEnv";
861 paths = [
862 # A Python 3 interpreter with some packages
863 (self.python3.withPackages (
864 ps: with ps; [
865 pyflakes
866 pytest
867 black
868 ]
869 ))
870
871 # Some other packages we'd like as part of this env
872 self.mypy
873 self.black
874 self.ripgrep
875 self.tmux
876 ];
877 };
878}
879```
880
881You can then build and install this to your profile with:
882
883```sh
884nix-env -iA myEnv
885```
886
887One limitation of this is that you can only have 1 Python env installed
888globally, since they conflict on the `python` to load out of your `PATH`.
889
890If you get a conflict or prefer to keep the setup clean, you can have `nix-env`
891atomically *uninstall* all other imperatively installed packages and replace
892your profile with just `myEnv` by using the `--replace` flag.
893
894##### Environment defined in `/etc/nixos/configuration.nix` {#environment-defined-in-etcnixosconfiguration.nix}
895
896For the sake of completeness, here's how to install the environment system-wide
897on NixOS.
898
899```nix
900{
901 # ...
902
903 environment.systemPackages = with pkgs; [
904 (python310.withPackages (
905 ps: with ps; [
906 numpy
907 toolz
908 ]
909 ))
910 ];
911}
912```
913
914### Developing with Python {#developing-with-python}
915
916Above, we were mostly just focused on use cases and what to do to get started
917creating working Python environments in nix.
918
919Now that you know the basics to be up and running, it is time to take a step
920back and take a deeper look at how Python packages are packaged on Nix.
921
922#### Python library packages in Nixpkgs {#python-library-packages-in-nixpkgs}
923
924With Nix all packages are built by functions. The main function in Nix for
925building Python libraries is [`buildPythonPackage`](#buildpythonpackage-function). Let's see how we can build the
926`toolz` package.
927
928```nix
929{
930 lib,
931 buildPythonPackage,
932 fetchPypi,
933 setuptools,
934}:
935
936buildPythonPackage rec {
937 pname = "toolz";
938 version = "0.10.0";
939 pyproject = true;
940
941 src = fetchPypi {
942 inherit pname version;
943 hash = "sha256-CP3V73yWSArRHBLUct4hrNMjWZlvaaUlkpm1QP66RWA=";
944 };
945
946 build-system = [
947 setuptools
948 ];
949
950 # has no tests
951 doCheck = false;
952
953 pythonImportsCheck = [
954 "toolz.itertoolz"
955 "toolz.functoolz"
956 "toolz.dicttoolz"
957 ];
958
959 meta = {
960 changelog = "https://github.com/pytoolz/toolz/releases/tag/${version}";
961 homepage = "https://github.com/pytoolz/toolz";
962 description = "List processing tools and functional utilities";
963 license = lib.licenses.bsd3;
964 };
965}
966```
967
968What happens here? The function [`buildPythonPackage`](#buildpythonpackage-function) is called and as argument
969it accepts a set. In this case the set is a recursive set, `rec`. One of the
970arguments is the name of the package, which consists of a basename (generally
971following the name on PyPI) and a version. Another argument, `src` specifies the
972source, which in this case is fetched from PyPI using the helper function
973`fetchPypi`. The argument `doCheck` is used to set whether tests should be run
974when building the package. Since there are no tests, we rely on [`pythonImportsCheck`](#using-pythonimportscheck)
975to test whether the package can be imported. Furthermore, we specify some meta
976information. The output of the function is a derivation.
977
978An expression for `toolz` can be found in the Nixpkgs repository. As explained
979in the introduction of this Python section, a derivation of `toolz` is available
980for each interpreter version, e.g. `python312.pkgs.toolz` refers to the `toolz`
981derivation corresponding to the CPython 3.12 interpreter.
982
983The above example works when you're directly working on
984`pkgs/top-level/python-packages.nix` in the Nixpkgs repository. Often though,
985you will want to test a Nix expression outside of the Nixpkgs tree.
986
987The following expression creates a derivation for the `toolz` package,
988and adds it along with a `numpy` package to a Python environment.
989
990```nix
991with import <nixpkgs> { };
992
993(
994 let
995 my_toolz = python312.pkgs.buildPythonPackage rec {
996 pname = "toolz";
997 version = "0.10.0";
998 pyproject = true;
999
1000 src = fetchPypi {
1001 inherit pname version;
1002 hash = "sha256-CP3V73yWSArRHBLUct4hrNMjWZlvaaUlkpm1QP66RWA=";
1003 };
1004
1005 build-system = [
1006 python312.pkgs.setuptools
1007 ];
1008
1009 # has no tests
1010 doCheck = false;
1011
1012 meta = {
1013 homepage = "https://github.com/pytoolz/toolz/";
1014 description = "List processing tools and functional utilities";
1015 # [...]
1016 };
1017 };
1018
1019 in
1020 python312.withPackages (
1021 ps: with ps; [
1022 numpy
1023 my_toolz
1024 ]
1025 )
1026).env
1027```
1028
1029Executing `nix-shell` will result in an environment in which you can use
1030Python 3.12 and the `toolz` package. As you can see we had to explicitly mention
1031for which Python version we want to build a package.
1032
1033So, what did we do here? Well, we took the Nix expression that we used earlier
1034to build a Python environment, and said that we wanted to include our own
1035version of `toolz`, named `my_toolz`. To introduce our own package in the scope
1036of [`withPackages`](#python.withpackages-function) we used a `let` expression. You can see that we used
1037`ps.numpy` to select numpy from the nixpkgs package set (`ps`). We did not take
1038`toolz` from the Nixpkgs package set this time, but instead took our own version
1039that we introduced with the `let` expression.
1040
1041#### Handling dependencies {#handling-dependencies}
1042
1043Our example, `toolz`, does not have any dependencies on other Python packages or system libraries.
1044[`buildPythonPackage`](#buildpythonpackage-function) uses the the following arguments in the following circumstances:
1045
1046- `dependencies` - For Python runtime dependencies.
1047- `build-system` - For Python build-time requirements.
1048- [`buildInputs`](#var-stdenv-buildInputs) - For non-Python build-time requirements.
1049- [`nativeCheckInputs`](#var-stdenv-nativeCheckInputs) - For test dependencies
1050
1051Dependencies can belong to multiple arguments, for example if something is both a build time requirement & a runtime dependency.
1052
1053The following example shows which arguments are given to [`buildPythonPackage`](#buildpythonpackage-function) in
1054order to build [`datashape`](https://github.com/blaze/datashape).
1055
1056```nix
1057{
1058 lib,
1059 buildPythonPackage,
1060 fetchPypi,
1061
1062 # build dependencies
1063 setuptools,
1064
1065 # dependencies
1066 numpy,
1067 multipledispatch,
1068 python-dateutil,
1069
1070 # tests
1071 pytestCheckHook,
1072}:
1073
1074buildPythonPackage rec {
1075 pname = "datashape";
1076 version = "0.4.7";
1077 pyproject = true;
1078
1079 src = fetchPypi {
1080 inherit pname version;
1081 hash = "sha256-FLLvdm1MllKrgTGC6Gb0k0deZeVYvtCCLji/B7uhong=";
1082 };
1083
1084 build-system = [
1085 setuptools
1086 ];
1087
1088 dependencies = [
1089 multipledispatch
1090 numpy
1091 python-dateutil
1092 ];
1093
1094 nativeCheckInputs = [
1095 pytestCheckHook
1096 ];
1097
1098 meta = {
1099 changelog = "https://github.com/blaze/datashape/releases/tag/${version}";
1100 homepage = "https://github.com/ContinuumIO/datashape";
1101 description = "Data description language";
1102 license = lib.licenses.bsd2;
1103 };
1104}
1105```
1106
1107We can see several runtime dependencies, `numpy`, `multipledispatch`, and
1108`python-dateutil`. Furthermore, we have [`nativeCheckInputs`](#var-stdenv-nativeCheckInputs) with `pytestCheckHook`.
1109`pytestCheckHook` is a test runner hook and is only used during the [`checkPhase`](#ssec-check-phase) and is
1110therefore not added to `dependencies`.
1111
1112In the previous case we had only dependencies on other Python packages to consider.
1113Occasionally you have also system libraries to consider. E.g., `lxml` provides
1114Python bindings to `libxml2` and `libxslt`. These libraries are only required
1115when building the bindings and are therefore added as [`buildInputs`](#var-stdenv-buildInputs).
1116
1117```nix
1118{
1119 lib,
1120 buildPythonPackage,
1121 fetchPypi,
1122 setuptools,
1123 libxml2,
1124 libxslt,
1125}:
1126
1127buildPythonPackage rec {
1128 pname = "lxml";
1129 version = "3.4.4";
1130 pyproject = true;
1131
1132 src = fetchPypi {
1133 inherit pname version;
1134 hash = "sha256-s9NiusRxFydHzaNRMjjxFcvWxfi45jGb9ql6eJJyQJk=";
1135 };
1136
1137 build-system = [
1138 setuptools
1139 ];
1140
1141 buildInputs = [
1142 libxml2
1143 libxslt
1144 ];
1145
1146 # tests are meant to be ran "in-place" in the same directory as src
1147 doCheck = false;
1148
1149 pythonImportsCheck = [
1150 "lxml"
1151 "lxml.etree"
1152 ];
1153
1154 meta = {
1155 changelog = "https://github.com/lxml/lxml/releases/tag/lxml-${version}";
1156 description = "Pythonic binding for the libxml2 and libxslt libraries";
1157 homepage = "https://lxml.de";
1158 license = lib.licenses.bsd3;
1159 maintainers = with lib.maintainers; [ sjourdois ];
1160 };
1161}
1162```
1163
1164In this example `lxml` and Nix are able to work out exactly where the relevant
1165files of the dependencies are. This is not always the case.
1166
1167The example below shows bindings to The Fastest Fourier Transform in the West,
1168commonly known as FFTW. On Nix we have separate packages of FFTW for the
1169different types of floats (`"single"`, `"double"`, `"long-double"`). The
1170bindings need all three types, and therefore we add all three as [`buildInputs`](#var-stdenv-buildInputs).
1171The bindings don't expect to find each of them in a different folder, and
1172therefore we have to set `LDFLAGS` and `CFLAGS`.
1173
1174```nix
1175{
1176 lib,
1177 buildPythonPackage,
1178 fetchPypi,
1179
1180 # build dependencies
1181 setuptools,
1182
1183 # dependencies
1184 fftw,
1185 fftwFloat,
1186 fftwLongDouble,
1187 numpy,
1188 scipy,
1189}:
1190
1191buildPythonPackage rec {
1192 pname = "pyfftw";
1193 version = "0.9.2";
1194 pyproject = true;
1195
1196 src = fetchPypi {
1197 inherit pname version;
1198 hash = "sha256-9ru2r6kwhUCaskiFoaPNuJCfCVoUL01J40byvRt4kHQ=";
1199 };
1200
1201 build-system = [
1202 setuptools
1203 ];
1204
1205 buildInputs = [
1206 fftw
1207 fftwFloat
1208 fftwLongDouble
1209 ];
1210
1211 dependencies = [
1212 numpy
1213 scipy
1214 ];
1215
1216 preConfigure = ''
1217 export LDFLAGS="-L${fftw.dev}/lib -L${fftwFloat.out}/lib -L${fftwLongDouble.out}/lib"
1218 export CFLAGS="-I${fftw.dev}/include -I${fftwFloat.dev}/include -I${fftwLongDouble.dev}/include"
1219 '';
1220
1221 # Tests cannot import pyfftw. pyfftw works fine though.
1222 doCheck = false;
1223
1224 pythonImportsCheck = [ "pyfftw" ];
1225
1226 meta = {
1227 changelog = "https://github.com/pyFFTW/pyFFTW/releases/tag/v${version}";
1228 description = "Pythonic wrapper around FFTW, the FFT library, presenting a unified interface for all the supported transforms";
1229 homepage = "http://hgomersall.github.com/pyFFTW";
1230 license = with lib.licenses; [
1231 bsd2
1232 bsd3
1233 ];
1234 };
1235}
1236```
1237
1238Note also the line [`doCheck = false;`](#var-stdenv-doCheck), we explicitly disabled running the test-suite.
1239
1240#### Testing Python Packages {#testing-python-packages}
1241
1242It is highly encouraged to have testing as part of the package build. This
1243helps to avoid situations where the package was able to build and install,
1244but is not usable at runtime.
1245Your package should provide its own [`checkPhase`](#ssec-check-phase).
1246
1247::: {.note}
1248The [`checkPhase`](#ssec-check-phase) for python maps to the `installCheckPhase` on a
1249normal derivation. This is due to many python packages not behaving well
1250to the pre-installed version of the package. Version info, and natively
1251compiled extensions generally only exist in the install directory, and
1252thus can cause issues when a test suite asserts on that behavior.
1253:::
1254
1255::: {.note}
1256Tests should only be disabled if they don't agree with nix
1257(e.g. external dependencies, network access, flakey tests), however,
1258as many tests should be enabled as possible. Failing tests can still be
1259a good indication that the package is not in a valid state.
1260:::
1261
1262#### Using pytest {#using-pytest}
1263
1264Pytest is the most common test runner for python repositories. A trivial
1265test run would be:
1266
1267```nix
1268{
1269 nativeCheckInputs = [ pytest ];
1270 checkPhase = ''
1271 runHook preCheck
1272
1273 pytest
1274
1275 runHook postCheck
1276 '';
1277}
1278```
1279
1280However, many repositories' test suites do not translate well to nix's build
1281sandbox, and will generally need many tests to be disabled.
1282
1283This is achievable by
1284- Including paths or test items (`path/to/file.py::MyClass` or `path/to/file.py::MyClass::test_method`) with positional arguments.
1285- Excluding paths with `--ignore` or globbed paths with `--ignore-glob`.
1286- Excluding test items using the `--deselect` flag.
1287- Including or excluding classes or test methods by their name using the `-k` flag.
1288- Including or excluding test by their marks using the `-m` flag.
1289
1290We highly recommend `pytestCheckHook` for an easier and more structural setup.
1291
1292#### Using pytestCheckHook {#using-pytestcheckhook}
1293
1294`pytestCheckHook` is a convenient hook which will set up (or configure)
1295a [`checkPhase`](#ssec-check-phase) to run `pytest`. This is also beneficial
1296when a package may need many items disabled to run the test suite.
1297Most packages use `pytest` or `unittest`, which is compatible with `pytest`,
1298so you will most likely use `pytestCheckHook`.
1299
1300To use `pytestCheckHook`, add it to `nativeCheckInputs`.
1301Adding `pytest` is not required, since it is included with `pytestCheckHook`.
1302
1303```nix
1304{
1305 nativeCheckInputs = [
1306 pytestCheckHook
1307 ];
1308}
1309```
1310
1311`pytestCheckHook` recognizes the following attributes:
1312
1313`enabledTestPaths` and `disabledTestPaths`
1314
1315: To specify path globs (files or directories) or test items.
1316
1317`enabledTests` and `disabledTests`
1318
1319: To specify keywords for class names or test method names.
1320
1321`enabledTestMarks` and `disabledTestMarks`
1322
1323: To specify test marks.
1324
1325`pytestFlags`
1326
1327: To append additional command-line arguments to `pytest`.
1328
1329By default, `pytest` automatically discovers which tests to run.
1330If tests are explicitly enabled, only those tests will run.
1331A test, that is both enabled and disabled, will not run.
1332
1333The following example demonstrates usage of various `pytestCheckHook` attributes:
1334
1335```nix
1336{
1337 nativeCheckInputs = [
1338 pytestCheckHook
1339 ];
1340
1341 # Allow running the following test paths and test objects.
1342 enabledTestPaths = [
1343 # Find tests under the tests directory.
1344 # The trailing slash is not necessary.
1345 "tests/"
1346
1347 # Additionally run test_foo
1348 "other-tests/test_foo.py::Foo::test_foo"
1349 ];
1350
1351 # Override the above-enabled test paths and test objects.
1352 disabledTestPaths = [
1353 # Tests under tests/integration requires additional data.
1354 "tests/integration"
1355 ];
1356
1357 # Allow tests by keywords matching their class names or method names.
1358 enabledTests = [
1359 # pytest by default only runs test methods begin with "test_" or end with "_test".
1360 # This includes all functions whose name contains "test".
1361 "test"
1362 ];
1363
1364 # Override the above-enabled tests by keywords matching their class names or method names.
1365 disabledTests = [
1366 # Tests touching networks.
1367 "upload"
1368 "download"
1369 ];
1370
1371 # Additional pytest flags
1372 pytestFlags = [
1373 # Disable benchmarks and run benchmarking tests only once.
1374 "--benchmark-disable"
1375 ];
1376}
1377```
1378
1379These attributes are all passed into the derivation directly
1380and added to the `pytest` command without additional Bash expansion.
1381It requires `__structuredAttrs = true` to pass list elements containing spaces.
1382
1383The `<enabled/disabled>TestsPaths` attributes expand Unix-style globs.
1384If a test path contains characters like `*`, `?`, `[`, or `]`, you can
1385quote them with square brackets (`[*]`, `[?]`, `[[]`, and `[]]`) to match literally.
1386
1387The `<enabled/disabled>Tests` and `<enabled/disabled>TestMarks` attribute pairs
1388form a logical expression `((included_element1) or (included_element2)) and not (excluded_element1) and not (excluded_element2)`
1389which will be passed to pytest's `-k` and `-m` flags respectively.
1390With `__structuredAttrs = true` enabled, they additionally support sub-expressions.
1391
1392For example, you could disable test items like `TestFoo::test_bar_functionality`
1393by disabling tests that match both `"Foo"` **and** `"bar"`:
1394
1395```nix
1396{
1397 __structuredAttrs = true;
1398
1399 disabledTests = [
1400 "Foo and bar"
1401 ];
1402}
1403```
1404
1405The main benefits of using `pytestCheckHook` to construct `pytest` commands
1406is structuralization and eval-time accessibility.
1407This is especially helpful to select tests or specify flags conditionally:
1408
1409```nix
1410{
1411 disabledTests =
1412 [
1413 # touches network
1414 "download"
1415 "update"
1416 ]
1417 ++ lib.optionals (pythonAtLeast "3.8") [
1418 # broken due to python3.8 async changes
1419 "async"
1420 ]
1421 ++ lib.optionals stdenv.buildPlatform.isDarwin [
1422 # can fail when building with other packages
1423 "socket"
1424 ];
1425}
1426```
1427
1428#### Using pythonImportsCheck {#using-pythonimportscheck}
1429
1430Although unit tests are highly preferred to validate correctness of a package, not
1431all packages have test suites that can be run easily, and some have none at all.
1432To help ensure the package still works, [`pythonImportsCheck`](#using-pythonimportscheck) can attempt to import
1433the listed modules.
1434
1435```nix
1436{
1437 pythonImportsCheck = [
1438 "requests"
1439 "urllib"
1440 ];
1441}
1442```
1443
1444roughly translates to:
1445
1446```nix
1447{
1448 postCheck = ''
1449 PYTHONPATH=$out/${python.sitePackages}:$PYTHONPATH
1450 python -c "import requests; import urllib"
1451 '';
1452}
1453```
1454
1455However, this is done in its own phase, and not dependent on whether [`doCheck = true;`](#var-stdenv-doCheck).
1456
1457This can also be useful in verifying that the package doesn't assume commonly
1458present packages (e.g. `setuptools`).
1459
1460#### Using pythonRelaxDepsHook {#using-pythonrelaxdepshook}
1461
1462It is common for upstream to specify a range of versions for its package
1463dependencies. This makes sense, since it ensures that the package will be built
1464with a subset of packages that is well tested. However, this commonly causes
1465issues when packaging in Nixpkgs, because the dependencies that this package
1466may need are too new or old for the package to build correctly. We also cannot
1467package multiple versions of the same package since this may cause conflicts
1468in `PYTHONPATH`.
1469
1470One way to side step this issue is to relax the dependencies. This can be done
1471by either removing the package version range or by removing the package
1472declaration entirely. This can be done using the `pythonRelaxDepsHook` hook. For
1473example, given the following `requirements.txt` file:
1474
1475```
1476pkg1<1.0
1477pkg2
1478pkg3>=1.0,<=2.0
1479```
1480
1481we can do:
1482
1483```nix
1484{
1485 pythonRelaxDeps = [
1486 "pkg1"
1487 "pkg3"
1488 ];
1489 pythonRemoveDeps = [
1490 "pkg2"
1491 ];
1492}
1493```
1494
1495which would result in the following `requirements.txt` file:
1496
1497```
1498pkg1
1499pkg3
1500```
1501
1502Another option is to pass `true`, that will relax/remove all dependencies, for
1503example:
1504
1505```nix
1506{
1507 pythonRelaxDeps = true;
1508}
1509```
1510
1511which would result in the following `requirements.txt` file:
1512
1513```
1514pkg1
1515pkg2
1516pkg3
1517```
1518
1519In general you should always use `pythonRelaxDeps`, because `pythonRemoveDeps`
1520will convert build errors into runtime errors. However `pythonRemoveDeps` may
1521still be useful in exceptional cases, and also to remove dependencies wrongly
1522declared by upstream (for example, declaring `black` as a runtime dependency
1523instead of a dev dependency).
1524
1525Keep in mind that while the examples above are done with `requirements.txt`,
1526`pythonRelaxDepsHook` works by modifying the resulting wheel file, so it should
1527work with any of the [existing hooks](#setup-hooks).
1528
1529The `pythonRelaxDepsHook` has no effect on build time dependencies, such as
1530those specified in `build-system`. If a package requires incompatible build
1531time dependencies, they should be removed in `postPatch` through
1532`substituteInPlace` or similar.
1533
1534For ease of use, both `buildPythonPackage` and `buildPythonApplication` will
1535automatically add `pythonRelaxDepsHook` if either `pythonRelaxDeps` or
1536`pythonRemoveDeps` is specified.
1537
1538#### Using unittestCheckHook {#using-unittestcheckhook}
1539
1540`unittestCheckHook` is a hook which will set up (or configure) a [`checkPhase`](#ssec-check-phase) to run `python -m unittest discover`:
1541
1542```nix
1543{
1544 nativeCheckInputs = [
1545 unittestCheckHook
1546 ];
1547
1548 unittestFlags = [
1549 "-s"
1550 "tests"
1551 "-v"
1552 ];
1553}
1554```
1555
1556`pytest` is compatible with `unittest`, so in most cases you can use `pytestCheckHook` instead.
1557
1558#### Using sphinxHook {#using-sphinxhook}
1559
1560The `sphinxHook` is a helpful tool to build documentation and manpages
1561using the popular Sphinx documentation generator.
1562It is setup to automatically find common documentation source paths and
1563render them using the default `html` style.
1564
1565```nix
1566{
1567 outputs = [
1568 "out"
1569 "doc"
1570 ];
1571
1572 nativeBuildInputs = [
1573 sphinxHook
1574 ];
1575}
1576```
1577
1578The hook will automatically build and install the artifact into the
1579`doc` output, if it exists. It also provides an automatic diversion
1580for the artifacts of the `man` builder into the `man` target.
1581
1582```nix
1583{
1584 outputs = [
1585 "out"
1586 "doc"
1587 "man"
1588 ];
1589
1590 # Use multiple builders
1591 sphinxBuilders = [
1592 "singlehtml"
1593 "man"
1594 ];
1595}
1596```
1597
1598Overwrite `sphinxRoot` when the hook is unable to find your
1599documentation source root.
1600
1601```nix
1602{
1603 # Configure sphinxRoot for uncommon paths
1604 sphinxRoot = "weird/docs/path";
1605}
1606```
1607
1608The hook is also available to packages outside the python ecosystem by
1609referencing it using `sphinxHook` from top-level.
1610
1611### Organising your packages {#organising-your-packages}
1612
1613So far we discussed how you can use Python on Nix, and how you can develop with
1614it. We've looked at how you write expressions to package Python packages, and we
1615looked at how you can create environments in which specified packages are
1616available.
1617
1618At some point you'll likely have multiple packages which you would
1619like to be able to use in different projects. In order to minimise unnecessary
1620duplication we now look at how you can maintain a repository with your
1621own packages. The important functions here are `import` and `callPackage`.
1622
1623### Including a derivation using `callPackage` {#including-a-derivation-using-callpackage}
1624
1625Earlier we created a Python environment using [`withPackages`](#python.withpackages-function), and included the
1626`toolz` package via a `let` expression.
1627Let's split the package definition from the environment definition.
1628
1629We first create a function that builds `toolz` in `~/path/to/toolz/release.nix`
1630
1631```nix
1632{
1633 lib,
1634 buildPythonPackage,
1635 fetchPypi,
1636 setuptools,
1637}:
1638
1639buildPythonPackage rec {
1640 pname = "toolz";
1641 version = "0.10.0";
1642 pyproject = true;
1643
1644 src = fetchPypi {
1645 inherit pname version;
1646 hash = "sha256-CP3V73yWSArRHBLUct4hrNMjWZlvaaUlkpm1QP66RWA=";
1647 };
1648
1649 build-system = [
1650 setuptools
1651 ];
1652
1653 meta = {
1654 changelog = "https://github.com/pytoolz/toolz/releases/tag/${version}";
1655 homepage = "https://github.com/pytoolz/toolz/";
1656 description = "List processing tools and functional utilities";
1657 license = lib.licenses.bsd3;
1658 };
1659}
1660```
1661
1662It takes an argument [`buildPythonPackage`](#buildpythonpackage-function). We now call this function using
1663`callPackage` in the definition of our environment
1664
1665```nix
1666with import <nixpkgs> { };
1667
1668(
1669 let
1670 toolz = callPackage /path/to/toolz/release.nix {
1671 buildPythonPackage = python3Packages.buildPythonPackage;
1672 };
1673 in
1674 python3.withPackages (ps: [
1675 ps.numpy
1676 toolz
1677 ])
1678).env
1679```
1680
1681Important to remember is that the Python version for which the package is made
1682depends on the `python` derivation that is passed to [`buildPythonPackage`](#buildpythonpackage-function). Nix
1683tries to automatically pass arguments when possible, which is why generally you
1684don't explicitly define which `python` derivation should be used. In the above
1685example we use [`buildPythonPackage`](#buildpythonpackage-function) that is part of the set `python3Packages`,
1686and in this case the `python3` interpreter is automatically used.
1687
1688## FAQ {#faq}
1689
1690### How to solve circular dependencies? {#how-to-solve-circular-dependencies}
1691
1692Consider the packages `A` and `B` that depend on each other. When packaging `B`,
1693a solution is to override package `A` not to depend on `B` as an input. The same
1694should also be done when packaging `A`.
1695
1696### How to override a Python package? {#how-to-override-a-python-package}
1697
1698We can override the interpreter and pass `packageOverrides`. In the following
1699example we rename the `pandas` package and build it.
1700
1701```nix
1702with import <nixpkgs> { };
1703
1704(
1705 let
1706 python =
1707 let
1708 packageOverrides = self: super: {
1709 pandas = super.pandas.overridePythonAttrs (old: {
1710 name = "foo";
1711 });
1712 };
1713 in
1714 pkgs.python310.override {
1715 inherit packageOverrides;
1716 };
1717
1718 in
1719 python.withPackages (ps: [
1720 ps.pandas
1721 ])
1722).env
1723```
1724
1725Using `nix-build` on this expression will build an environment that contains the
1726package `pandas` but with the new name `foo`.
1727
1728All packages in the package set will use the renamed package. A typical use case
1729is to switch to another version of a certain package. For example, in the
1730Nixpkgs repository we have multiple versions of `django` and `scipy`. In the
1731following example we use a different version of `scipy` and create an
1732environment that uses it. All packages in the Python package set will now use
1733the updated `scipy` version.
1734
1735```nix
1736with import <nixpkgs> { };
1737
1738(
1739 let
1740 packageOverrides = self: super: {
1741 scipy = super.scipy_0_17;
1742 };
1743 in
1744 (pkgs.python310.override {
1745 inherit packageOverrides;
1746 }).withPackages
1747 (ps: [
1748 ps.blaze
1749 ])
1750).env
1751```
1752
1753The requested package `blaze` depends on `pandas` which itself depends on `scipy`.
1754
1755If you want the whole of Nixpkgs to use your modifications, then you can use
1756`overlays` as explained in this manual. In the following example we build a
1757`inkscape` using a different version of `numpy`.
1758
1759```nix
1760let
1761 pkgs = import <nixpkgs> { };
1762 newpkgs = import pkgs.path {
1763 overlays = [
1764 (self: super: {
1765 python310 =
1766 let
1767 packageOverrides = python-self: python-super: {
1768 numpy = python-super.numpy_1_18;
1769 };
1770 in
1771 super.python310.override { inherit packageOverrides; };
1772 })
1773 ];
1774 };
1775in
1776newpkgs.inkscape
1777```
1778
1779### `python setup.py bdist_wheel` cannot create .whl {#python-setup.py-bdist_wheel-cannot-create-.whl}
1780
1781Executing `python setup.py bdist_wheel` in a `nix-shell`fails with
1782
1783```
1784ValueError: ZIP does not support timestamps before 1980
1785```
1786
1787This is because files from the Nix store (which have a timestamp of the UNIX
1788epoch of January 1, 1970) are included in the .ZIP, but .ZIP archives follow the
1789DOS convention of counting timestamps from 1980.
1790
1791The command `bdist_wheel` reads the `SOURCE_DATE_EPOCH` environment variable,
1792which `nix-shell` sets to 1. Unsetting this variable or giving it a value
1793corresponding to 1980 or later enables building wheels.
1794
1795Use 1980 as timestamp:
1796
1797```shell
1798nix-shell --run "SOURCE_DATE_EPOCH=315532800 python3 setup.py bdist_wheel"
1799```
1800
1801or the current time:
1802
1803```shell
1804nix-shell --run "SOURCE_DATE_EPOCH=$(date +%s) python3 setup.py bdist_wheel"
1805```
1806
1807or unset `SOURCE_DATE_EPOCH`:
1808
1809```shell
1810nix-shell --run "unset SOURCE_DATE_EPOCH; python3 setup.py bdist_wheel"
1811```
1812
1813### `install_data` / `data_files` problems {#install_data-data_files-problems}
1814
1815If you get the following error:
1816
1817```
1818could not create '/nix/store/6l1bvljpy8gazlsw2aw9skwwp4pmvyxw-python-2.7.8/etc':
1819Permission denied
1820```
1821
1822This is a [known bug](https://github.com/pypa/setuptools/issues/130) in
1823`setuptools`. Setuptools `install_data` does not respect `--prefix`. An example
1824of such package using the feature is `pkgs/tools/X11/xpra/default.nix`.
1825
1826As workaround install it as an extra `preInstall` step:
1827
1828```shell
1829${python.pythonOnBuildForHost.interpreter} setup.py install_data --install-dir=$out --root=$out
1830sed -i '/ = data\_files/d' setup.py
1831```
1832
1833### Rationale of non-existent global site-packages {#rationale-of-non-existent-global-site-packages}
1834
1835On most operating systems a global `site-packages` is maintained. This however
1836becomes problematic if you want to run multiple Python versions or have multiple
1837versions of certain libraries for your projects. Generally, you would solve such
1838issues by creating virtual environments using `virtualenv`.
1839
1840On Nix each package has an isolated dependency tree which, in the case of
1841Python, guarantees the right versions of the interpreter and libraries or
1842packages are available. There is therefore no need to maintain a global `site-packages`.
1843
1844If you want to create a Python environment for development, then the recommended
1845method is to use `nix-shell`, either with or without the [`python.buildEnv`](#python.buildenv-function)
1846function.
1847
1848### How to consume Python modules using pip in a virtual environment like I am used to on other Operating Systems? {#how-to-consume-python-modules-using-pip-in-a-virtual-environment-like-i-am-used-to-on-other-operating-systems}
1849
1850While this approach is not very idiomatic from Nix perspective, it can still be
1851useful when dealing with pre-existing projects or in situations where it's not
1852feasible or desired to write derivations for all required dependencies.
1853
1854This is an example of a `default.nix` for a `nix-shell`, which allows to consume
1855a virtual environment created by `venv`, and install Python modules through
1856`pip` the traditional way.
1857
1858Create this `default.nix` file, together with a `requirements.txt` and
1859execute `nix-shell`.
1860
1861```nix
1862with import <nixpkgs> { };
1863
1864let
1865 pythonPackages = python3Packages;
1866in
1867pkgs.mkShell rec {
1868 name = "impurePythonEnv";
1869 venvDir = "./.venv";
1870 buildInputs = [
1871 # A Python interpreter including the 'venv' module is required to bootstrap
1872 # the environment.
1873 pythonPackages.python
1874
1875 # This executes some shell code to initialize a venv in $venvDir before
1876 # dropping into the shell
1877 pythonPackages.venvShellHook
1878
1879 # Those are dependencies that we would like to use from nixpkgs, which will
1880 # add them to PYTHONPATH and thus make them accessible from within the venv.
1881 pythonPackages.numpy
1882 pythonPackages.requests
1883
1884 # In this particular example, in order to compile any binary extensions they may
1885 # require, the Python modules listed in the hypothetical requirements.txt need
1886 # the following packages to be installed locally:
1887 taglib
1888 openssl
1889 git
1890 libxml2
1891 libxslt
1892 libzip
1893 zlib
1894 ];
1895
1896 # Run this command, only after creating the virtual environment
1897 postVenvCreation = ''
1898 unset SOURCE_DATE_EPOCH
1899 pip install -r requirements.txt
1900 '';
1901
1902 # Now we can execute any commands within the virtual environment.
1903 # This is optional and can be left out to run pip manually.
1904 postShellHook = ''
1905 # allow pip to install wheels
1906 unset SOURCE_DATE_EPOCH
1907 '';
1908
1909}
1910```
1911
1912In case the supplied venvShellHook is insufficient, or when Python 2 support is
1913needed, you can define your own shell hook and adapt to your needs like in the
1914following example:
1915
1916```nix
1917with import <nixpkgs> { };
1918
1919let
1920 venvDir = "./.venv";
1921 pythonPackages = python3Packages;
1922in
1923pkgs.mkShell rec {
1924 name = "impurePythonEnv";
1925 buildInputs = [
1926 pythonPackages.python
1927 # Needed when using python 2.7
1928 # pythonPackages.virtualenv
1929 # ...
1930 ];
1931
1932 # This is very close to how venvShellHook is implemented, but
1933 # adapted to use 'virtualenv'
1934 shellHook = ''
1935 SOURCE_DATE_EPOCH=$(date +%s)
1936
1937 if [ -d "${venvDir}" ]; then
1938 echo "Skipping venv creation, '${venvDir}' already exists"
1939 else
1940 echo "Creating new venv environment in path: '${venvDir}'"
1941 # Note that the module venv was only introduced in python 3, so for 2.7
1942 # this needs to be replaced with a call to virtualenv
1943 ${pythonPackages.python.interpreter} -m venv "${venvDir}"
1944 fi
1945
1946 # Under some circumstances it might be necessary to add your virtual
1947 # environment to PYTHONPATH, which you can do here too;
1948 # PYTHONPATH=$PWD/${venvDir}/${pythonPackages.python.sitePackages}/:$PYTHONPATH
1949
1950 source "${venvDir}/bin/activate"
1951
1952 # As in the previous example, this is optional.
1953 pip install -r requirements.txt
1954 '';
1955}
1956```
1957
1958Note that the `pip install` is an imperative action. So every time `nix-shell`
1959is executed it will attempt to download the Python modules listed in
1960requirements.txt. However these will be cached locally within the `virtualenv`
1961folder and not downloaded again.
1962
1963### How to override a Python package from `configuration.nix`? {#how-to-override-a-python-package-from-configuration.nix}
1964
1965If you need to change a package's attribute(s) from `configuration.nix` you could do:
1966
1967```nix
1968{
1969 nixpkgs.config.packageOverrides = super: {
1970 python3 = super.python3.override {
1971 packageOverrides = python-self: python-super: {
1972 twisted = python-super.twisted.overridePythonAttrs (oldAttrs: {
1973 src = super.fetchPypi {
1974 pname = "Twisted";
1975 version = "19.10.0";
1976 hash = "sha256-c5S6fycq5yKnTz2Wnc9Zm8TvCTvDkgOHSKSQ8XJKUV0=";
1977 extension = "tar.bz2";
1978 };
1979 });
1980 };
1981 };
1982 };
1983}
1984```
1985
1986`python3Packages.twisted` is now globally overridden.
1987All packages and also all NixOS services that reference `twisted`
1988(such as `services.buildbot-worker`) now use the new definition.
1989Note that `python-super` refers to the old package set and `python-self`
1990to the new, overridden version.
1991
1992To modify only a Python package set instead of a whole Python derivation, use
1993this snippet:
1994
1995```nix
1996{
1997 myPythonPackages = python3Packages.override {
1998 overrides = self: super: {
1999 twisted = <...>;
2000 };
2001 };
2002}
2003```
2004
2005### How to override a Python package using overlays? {#how-to-override-a-python-package-using-overlays}
2006
2007Use the following overlay template:
2008
2009```nix
2010self: super: {
2011 python = super.python.override {
2012 packageOverrides = python-self: python-super: {
2013 twisted = python-super.twisted.overrideAttrs (oldAttrs: {
2014 src = super.fetchPypi {
2015 pname = "Twisted";
2016 version = "19.10.0";
2017 hash = "sha256-c5S6fycq5yKnTz2Wnc9Zm8TvCTvDkgOHSKSQ8XJKUV0=";
2018 extension = "tar.bz2";
2019 };
2020 });
2021 };
2022 };
2023}
2024```
2025
2026### How to override a Python package for all Python versions using extensions? {#how-to-override-a-python-package-for-all-python-versions-using-extensions}
2027
2028The following overlay overrides the call to [`buildPythonPackage`](#buildpythonpackage-function) for the
2029`foo` package for all interpreters by appending a Python extension to the
2030`pythonPackagesExtensions` list of extensions.
2031
2032```nix
2033final: prev: {
2034 pythonPackagesExtensions = prev.pythonPackagesExtensions ++ [
2035 (python-final: python-prev: {
2036 foo = python-prev.foo.overridePythonAttrs (oldAttrs: {
2037 # ...
2038 });
2039 })
2040 ];
2041}
2042```
2043
2044### How to use Intel’s MKL with numpy and scipy? {#how-to-use-intels-mkl-with-numpy-and-scipy}
2045
2046MKL can be configured using an overlay. See the section "[Using overlays to
2047configure alternatives](#sec-overlays-alternatives-blas-lapack)".
2048
2049### What inputs do `setup_requires`, `install_requires` and `tests_require` map to? {#what-inputs-do-setup_requires-install_requires-and-tests_require-map-to}
2050
2051In a `setup.py` or `setup.cfg` it is common to declare dependencies:
2052
2053* `setup_requires` corresponds to `build-system`
2054* `install_requires` corresponds to `dependencies`
2055* `tests_require` corresponds to [`nativeCheckInputs`](#var-stdenv-nativeCheckInputs)
2056
2057### How to enable interpreter optimizations? {#optimizations}
2058
2059The Python interpreters are by default not built with optimizations enabled, because
2060the builds are in that case not reproducible. To enable optimizations, override the
2061interpreter of interest, e.g using
2062
2063```nix
2064let
2065 pkgs = import ./. { };
2066 mypython = pkgs.python3.override {
2067 enableOptimizations = true;
2068 reproducibleBuild = false;
2069 self = mypython;
2070 };
2071in
2072mypython
2073```
2074
2075### How to add optional dependencies? {#python-optional-dependencies}
2076
2077Some packages define optional dependencies for additional features. With
2078`setuptools` this is called `extras_require` and `flit` calls it
2079`extras-require`, while PEP 621 calls these `optional-dependencies`.
2080
2081```nix
2082{
2083 optional-dependencies = {
2084 complete = [ distributed ];
2085 };
2086}
2087```
2088
2089and letting the package requiring the extra add the list to its dependencies
2090
2091```nix
2092{
2093 dependencies = [
2094 # ...
2095 ] ++ dask.optional-dependencies.complete;
2096}
2097```
2098
2099This method is using `passthru`, meaning that changing `optional-dependencies` of a package won't cause it to rebuild.
2100
2101Note this method is preferred over adding parameters to builders, as that can
2102result in packages depending on different variants and thereby causing
2103collisions.
2104
2105::: {.note}
2106The `optional-dependencies` attribute should only be used for dependency groups
2107as defined in package metadata. If a package gracefully handles missing
2108dependencies in runtime but doesn't advertise it through package metadata, then
2109these dependencies should not be listed at all. (One may still have to list
2110them in `nativeCheckInputs` to pass test suite.)
2111:::
2112
2113### How to contribute a Python package to nixpkgs? {#tools}
2114
2115Packages inside nixpkgs must use the [`buildPythonPackage`](#buildpythonpackage-function) or [`buildPythonApplication`](#buildpythonapplication-function) function directly,
2116because we can only provide security support for non-vendored dependencies.
2117
2118We recommend [nix-init](https://github.com/nix-community/nix-init) for creating new python packages within nixpkgs,
2119as it already prefetches the source, parses dependencies for common formats and prefills most things in `meta`.
2120When using the tool, pull from the original source repository instead of PyPI, if possible.
2121
2122See also [contributing section](#contributing).
2123
2124### Are Python interpreters built deterministically? {#deterministic-builds}
2125
2126The Python interpreters are now built deterministically. Minor modifications had
2127to be made to the interpreters in order to generate deterministic bytecode. This
2128has security implications and is relevant for those using Python in a
2129`nix-shell`.
2130
2131When the environment variable `DETERMINISTIC_BUILD` is set, all bytecode will
2132have timestamp 1. The [`buildPythonPackage`](#buildpythonpackage-function) function sets `DETERMINISTIC_BUILD=1`
2133and [PYTHONHASHSEED=0](https://docs.python.org/3.12/using/cmdline.html#envvar-PYTHONHASHSEED).
2134Both are also exported in `nix-shell`.
2135
2136### How to provide automatic tests to Python packages? {#automatic-tests}
2137
2138It is recommended to test packages as part of the build process.
2139Source distributions (`sdist`) often include test files, but not always.
2140
2141The best practice today is to pass a test hook (e.g. pytestCheckHook, unittestCheckHook) into nativeCheckInputs.
2142This will reconfigure the checkPhase to make use of that particular test framework.
2143Occasionally packages don't make use of a common test framework, which may then require a custom checkPhase.
2144
2145#### Common issues {#common-issues}
2146
2147* Tests that attempt to access `$HOME` can be fixed by using `writableTmpDirAsHomeHook` in
2148 `nativeCheckInputs`, which sets up a writable temporary directory as the home directory. Alternatively,
2149 you can achieve the same effect manually (e.g. in `preCheck`) with: `export HOME=$(mktemp -d)`.
2150* Compiling with Cython causes tests to fail with a `ModuleNotLoadedError`.
2151 This can be fixed with two changes in the derivation: 1) replacing `pytest` with
2152 `pytestCheckHook` and 2) adding a `preCheck` containing `cd $out` to run
2153 tests within the built output.
2154
2155## Contributing {#contributing}
2156
2157### Contributing guidelines {#contributing-guidelines}
2158
2159The following rules are desired to be respected:
2160
2161* Python libraries are called from `python-packages.nix` and packaged with
2162 [`buildPythonPackage`](#buildpythonpackage-function). The expression of a library should be in
2163 `pkgs/development/python-modules/<name>/default.nix`.
2164* Python applications live outside of `python-packages.nix` and are packaged
2165 with [`buildPythonApplication`](#buildpythonapplication-function).
2166* Make sure libraries build for all Python interpreters.
2167 If it fails to build on some Python versions, consider disabling them by setting `disable = pythonAtLeast "3.x"` along with a comment.
2168* The two parameters, `pyproject` and `build-system` are set to avoid the legacy setuptools/distutils build.
2169* Only unversioned attributes (e.g. `pydantic`, but not `pypdantic_1`) can be included in `dependencies`,
2170 since due to `PYTHONPATH` limitations we can only ever support a single version for libraries
2171 without running into duplicate module name conflicts.
2172* The version restrictions of `dependencies` can be relaxed by [`pythonRelaxDepsHook`](#using-pythonrelaxdepshook).
2173* Make sure the tests are enabled using for example [`pytestCheckHook`](#using-pytestcheckhook) and, in the case of
2174 libraries, are passing for all interpreters. If certain tests fail they can be
2175 disabled individually. Try to avoid disabling the tests altogether. In any
2176 case, when you disable tests, leave a comment explaining why.
2177* `pythonImportsCheck` is set. This is still a good smoke test even if `pytestCheckHook` is set.
2178* `meta.platforms` takes the default value in many cases.
2179 It does not need to be set explicitly unless the package requires a specific platform.
2180* The file is formatted with `nixfmt-rfc-style`.
2181* Commit names of Python libraries must reflect that they are Python
2182 libraries (e.g. `python312Packages.numpy: 1.11 -> 1.12` rather than `numpy: 1.11 -> 1.12`).
2183* The current default version of python should be included
2184 in commit messages to enable automatic builds by ofborg.
2185 For example `python312Packages.numpy: 1.11 -> 1.12` should be used rather
2186 than `python3Packages.numpy: 1.11 -> 1.12`.
2187 Note that `pythonPackages` is an alias for `python27Packages`.
2188* Attribute names in `python-packages.nix` as well as `pname`s should match the
2189 library's name on PyPI, but be normalized according to [PEP
2190 0503](https://www.python.org/dev/peps/pep-0503/#normalized-names). This means
2191 that characters should be converted to lowercase and `.` and `_` should be
2192 replaced by a single `-` (foo-bar-baz instead of Foo__Bar.baz).
2193 If necessary, `pname` has to be given a different value within `fetchPypi`.
2194* It's generally preferable to fetch `src` directly from the repo and not from
2195 PyPI. Use `fetchPypi` when there's a clear technical reason to do so.
2196* Packages from sources such as GitHub and GitLab that do not exist on PyPI
2197 should not use a name that is already used on PyPI. When possible, they should
2198 use the package repository name prefixed with the owner (e.g. organization) name
2199 and using a `-` as delimiter.
2200* Attribute names in `python-packages.nix` should be sorted alphanumerically to
2201 avoid merge conflicts and ease locating attributes.
2202* Non-python runtime dependencies should be added via explicit wrapping or
2203 patching (using e.g. `substituteInPlace`), rather than through propagation via
2204 `dependencies`/`propagatedBuildInputs`, to reduce clutter in `$PATH`.
2205
2206This list is useful for reviewers as well as for self-checking when submitting packages.
2207
2208## Package set maintenance {#python-package-set-maintenance}
2209
2210The whole Python package set has a lot of packages that do not see regular
2211updates, because they either are a very fragile component in the Python
2212ecosystem, like for example the `hypothesis` package, or packages that have
2213no maintainer, so maintenance falls back to the package set maintainers.
2214
2215### Updating packages in bulk {#python-package-bulk-updates}
2216
2217A tool to bulk-update numerous Python libraries is available in the
2218repository at `maintainers/scripts/update-python-libraries`.
2219
2220It can quickly update minor or major versions for all packages selected
2221and create update commits, and supports the `fetchPypi`, `fetchurl` and
2222`fetchFromGitHub` fetchers. When updating lots of packages that are
2223hosted on GitHub, exporting a `GITHUB_API_TOKEN` is highly recommended.
2224
2225Updating packages in bulk leads to lots of breakages, which is why a
2226stabilization period on the `python-updates` branch is required.
2227
2228If a package is fragile and often breaks during these bulks updates, it
2229may be reasonable to set `passthru.skipBulkUpdate = true` in the
2230derivation. This decision should not be made on a whim and should
2231always be supported by a qualifying comment.
2232
2233Once the branch is sufficiently stable it should normally be merged
2234into the `staging` branch.
2235
2236An exemplary call to update all python libraries between minor versions
2237would be:
2238
2239```ShellSession
2240$ maintainers/scripts/update-python-libraries --target minor --commit --use-pkgs-prefix pkgs/development/python-modules/**/default.nix
2241```
2242
2243## CPython Update Schedule {#python-cpython-update-schedule}
2244
2245With [PEP 602](https://www.python.org/dev/peps/pep-0602/), CPython now
2246follows a yearly release cadence. In nixpkgs, all supported interpreters
2247are made available, but only the most recent two
2248interpreters package sets are built; this is a compromise between being
2249the latest interpreter, and what the majority of the Python packages support.
2250
2251New CPython interpreters are released in October. Generally, it takes some
2252time for the majority of active Python projects to support the latest stable
2253interpreter. To help ease the migration for Nixpkgs users
2254between Python interpreters the schedule below will be used:
2255
2256| When | Event |
2257| --- | --- |
2258| After YY.11 Release | Bump CPython package set window. The latest and previous latest stable should now be built. |
2259| After YY.05 Release | Bump default CPython interpreter to latest stable. |
2260
2261In practice, this means that the Python community will have had a stable interpreter
2262for ~2 months before attempting to update the package set. And this will
2263allow for ~7 months for Python applications to support the latest interpreter.