pyrox.dev
1# Python {#python}
2
3## Reference {#reference}
4
5### Interpreters {#interpreters}
6
7@python-interpreter-table@
8
9The Nix expressions for the interpreters can be found in
10`pkgs/development/interpreters/python`.
11
12All packages depending on any Python interpreter get appended
13`out/{python.sitePackages}` to `$PYTHONPATH` if such directory
14exists.
15
16#### Missing `tkinter` module standard library {#missing-tkinter-module-standard-library}
17
18To reduce closure size the `Tkinter`/`tkinter` is available as a separate package, `pythonPackages.tkinter`.
19
20#### Attributes on interpreters packages {#attributes-on-interpreters-packages}
21
22Each interpreter has the following attributes:
23
24- `libPrefix`. Name of the folder in `${python}/lib/` for corresponding interpreter.
25- `interpreter`. Alias for `${python}/bin/${executable}`.
26- `buildEnv`. Function to build python interpreter environments with extra packages bundled together. See [](#python.buildenv-function) for usage and documentation.
27- `withPackages`. Simpler interface to `buildEnv`. See [](#python.withpackages-function) for usage and documentation.
28- `sitePackages`. Alias for `lib/${libPrefix}/site-packages`.
29- `executable`. Name of the interpreter executable, e.g. `python3.10`.
30- `pkgs`. Set of Python packages for that specific interpreter. The package set can be modified by overriding the interpreter and passing `packageOverrides`.
31
32### Building packages and applications {#building-packages-and-applications}
33
34Python libraries and applications that use tools to follow PEP 517 (e.g. `setuptools` or `hatchling`, etc.) or
35previous tools such as `distutils` are typically built with respectively the [`buildPythonPackage`](#buildpythonpackage-function) and
36[`buildPythonApplication`](#buildpythonapplication-function) functions. These two functions also support installing a `wheel`.
37
38All Python packages reside in `pkgs/top-level/python-packages.nix` and all
39applications elsewhere. In case a package is used as both a library and an
40application, then the package should be in `pkgs/top-level/python-packages.nix`
41since only those packages are made available for all interpreter versions. The
42preferred location for library expressions is in
43`pkgs/development/python-modules`. It is important that these packages are
44called from `pkgs/top-level/python-packages.nix` and not elsewhere, to guarantee
45the right version of the package is built.
46
47Based on the packages defined in `pkgs/top-level/python-packages.nix` an
48attribute set is created for each available Python interpreter. The available
49sets are
50
51* `pkgs.python27Packages`
52* `pkgs.python3Packages`
53* `pkgs.python310Packages`
54* `pkgs.python311Packages`
55* `pkgs.python312Packages`
56* `pkgs.python313Packages`
57* `pkgs.python314Packages`
58* `pkgs.pypy27Packages`
59* `pkgs.pypy310Packages`
60
61and the aliases
62
63* `pkgs.python2Packages` pointing to `pkgs.python27Packages`
64* `pkgs.python3Packages` pointing to `pkgs.python313Packages`
65* `pkgs.pythonPackages` pointing to `pkgs.python2Packages`
66* `pkgs.pypy2Packages` pointing to `pkgs.pypy27Packages`
67* `pkgs.pypy3Packages` pointing to `pkgs.pypy310Packages`
68* `pkgs.pypyPackages` pointing to `pkgs.pypy2Packages`
69
70
71#### `buildPythonPackage` function {#buildpythonpackage-function}
72
73The `buildPythonPackage` function has its name binding in
74`pkgs/development/interpreters/python/python-packages-base.nix` and is
75implemented in `pkgs/development/interpreters/python/mk-python-derivation.nix`
76using setup hooks.
77
78The following is an example:
79
80```nix
81{
82 lib,
83 buildPythonPackage,
84 fetchPypi,
85
86 # build-system
87 setuptools,
88 setuptools-scm,
89
90 # dependencies
91 attrs,
92 pluggy,
93 py,
94 setuptools,
95 six,
96
97 # tests
98 hypothesis,
99}:
100
101buildPythonPackage rec {
102 pname = "pytest";
103 version = "3.3.1";
104 pyproject = true;
105
106 src = fetchPypi {
107 inherit pname version;
108 hash = "sha256-z4Q23FnYaVNG/NOrKW3kZCXsqwDWQJbOvnn7Ueyy65M=";
109 };
110
111 postPatch = ''
112 # don't test bash builtins
113 rm testing/test_argcomplete.py
114 '';
115
116 build-system = [
117 setuptools
118 setuptools-scm
119 ];
120
121 dependencies = [
122 attrs
123 py
124 setuptools
125 six
126 pluggy
127 ];
128
129 nativeCheckInputs = [ hypothesis ];
130
131 meta = {
132 changelog = "https://github.com/pytest-dev/pytest/releases/tag/${version}";
133 description = "Framework for writing tests";
134 homepage = "https://github.com/pytest-dev/pytest";
135 license = lib.licenses.mit;
136 maintainers = with lib.maintainers; [
137 lovek323
138 madjar
139 lsix
140 ];
141 };
142}
143```
144
145The `buildPythonPackage` mainly does four things:
146
147* In the [`buildPhase`](#build-phase), it calls `${python.pythonOnBuildForHost.interpreter} -m build --wheel` to
148 build a wheel binary zipfile.
149* In the [`installPhase`](#ssec-install-phase), it installs the wheel file using `${python.pythonOnBuildForHost.interpreter} -m installer *.whl`.
150* In the [`postFixup`](#var-stdenv-postFixup) phase, the `wrapPythonPrograms` bash function is called to
151 wrap all programs in the `$out/bin/*` directory to include `$PATH`
152 environment variable and add dependent libraries to script's `sys.path`.
153* In the [`installCheck`](#ssec-installCheck-phase) phase, `${python.interpreter} -m pytest` is run.
154
155By default tests are run because [`doCheck = true`](#var-stdenv-doCheck). Test dependencies, like
156e.g. the test runner, should be added to [`nativeCheckInputs`](#var-stdenv-nativeCheckInputs).
157
158By default `meta.platforms` is set to the same value
159as the interpreter unless overridden otherwise.
160
161##### `buildPythonPackage` parameters {#buildpythonpackage-parameters}
162
163All parameters from [`stdenv.mkDerivation`](#sec-using-stdenv) function are still supported. The
164following are specific to `buildPythonPackage`:
165
166* `catchConflicts ? true`: If `true`, abort package build if a package name
167 appears more than once in dependency tree. Default is `true`.
168* `disabled ? false`: If `true`, package is not built for the particular Python
169 interpreter version.
170* `dontWrapPythonPrograms ? false`: Skip wrapping of Python programs.
171* `permitUserSite ? false`: Skip setting the `PYTHONNOUSERSITE` environment
172 variable in wrapped programs.
173* `pyproject`: Whether the pyproject format should be used. As all other formats
174 are deprecated, you are recommended to set this to `true`. When you do so,
175 `pypaBuildHook` will be used, and you can add the required build dependencies
176 from `build-system.requires` to `build-system`. Note that the pyproject
177 format falls back to using `setuptools`, so you can use `pyproject = true`
178 even if the package only has a `setup.py`. When set to `false`, you can
179 use the existing [hooks](#setup-hooks) or provide your own logic to build the
180 package. This can be useful for packages that don't support the pyproject
181 format. When unset, the legacy `setuptools` hooks are used for backwards
182 compatibility.
183* `makeWrapperArgs ? []`: A list of strings. Arguments to be passed to
184 [`makeWrapper`](#fun-makeWrapper), which wraps generated binaries. By default, the arguments to
185 [`makeWrapper`](#fun-makeWrapper) set `PATH` and `PYTHONPATH` environment variables before calling
186 the binary. Additional arguments here can allow a developer to set environment
187 variables which will be available when the binary is run. For example,
188 `makeWrapperArgs = ["--set" "FOO" "BAR" "--set" "BAZ" "QUX"]`.
189
190 ::: {.note}
191 When `__structuredAttrs = false`, the attribute `makeWrapperArgs` is passed as a space-separated string to the build script. Developers should use `prependToVar` or `appendToVar` to add arguments to it in build phases, or use `__structuredAttrs = true` to ensure that `makeWrapperArgs` is passed as a Bash array.
192
193 For compatibility purposes,
194 when `makeWrapperArgs` shell variable is specified as a space-separated string (instead of a Bash array) in the build script, the string content is Bash-expanded before concatenated into the `wrapProgram` command. Still, developers should not rely on such behaviours, but use `__structuredAttrs = true` to specify flags containing spaces (e.g. `makeWrapperArgs = [ "--set" "GREETING" "Hello, world!" ]`), or use -pre and -post phases to specify flags with Bash-expansions (e.g. `preFixup = ''makeWrapperArgs+=(--prefix PATH : "$SOME_PATH")`'').
195 :::
196
197* `namePrefix`: Prepends text to `${name}` parameter. In case of libraries, this
198 defaults to `"python3.8-"` for Python 3.8, etc., and in case of applications to `""`.
199* `pypaBuildFlags ? []`: A list of strings. Arguments to be passed to `python -m build --wheel`.
200* `pythonPath ? []`: List of packages to be added into `$PYTHONPATH`. Packages
201 in `pythonPath` are not propagated (contrary to [`propagatedBuildInputs`](#var-stdenv-propagatedBuildInputs)).
202* `preShellHook`: Hook to execute commands before `shellHook`.
203* `postShellHook`: Hook to execute commands after `shellHook`.
204* `removeBinByteCode ? true`: Remove bytecode from `/bin`. Bytecode is only
205 created when the filenames end with `.py`.
206* `setupPyGlobalFlags ? []`: List of flags passed to `setup.py` command.
207* `setupPyBuildFlags ? []`: List of flags passed to `setup.py build_ext` command.
208
209The [`stdenv.mkDerivation`](#sec-using-stdenv) function accepts various parameters for describing
210build inputs (see "Specifying dependencies"). The following are of special
211interest for Python packages, either because these are primarily used, or
212because their behaviour is different:
213
214* `nativeBuildInputs ? []`: Build-time only dependencies. Typically executables.
215* `build-system ? []`: Build-time only Python dependencies. Items listed in `build-system.requires`/`setup_requires`.
216* `buildInputs ? []`: Build and/or run-time dependencies that need to be
217 compiled for the host machine. Typically non-Python libraries which are being
218 linked.
219* `nativeCheckInputs ? []`: Dependencies needed for running the [`checkPhase`](#ssec-check-phase). These
220 are added to [`nativeBuildInputs`](#var-stdenv-nativeBuildInputs) when [`doCheck = true`](#var-stdenv-doCheck). Items listed in
221 `tests_require` go here.
222* `dependencies ? []`: Aside from propagating dependencies,
223 `buildPythonPackage` also injects code into and wraps executables with the
224 paths included in this list. Items listed in `install_requires` go here.
225* `optional-dependencies ? { }`: Optional feature flagged dependencies. Items listed in `extras_require` go here.
226
227
228##### Overriding Python packages {#overriding-python-packages}
229
230The `buildPythonPackage` function has a `overridePythonAttrs` method that can be
231used to override the package. In the following example we create an environment
232where we have the `blaze` package using an older version of `pandas`. We
233override first the Python interpreter and pass `packageOverrides` which contains
234the overrides for packages in the package set.
235
236```nix
237with import <nixpkgs> { };
238
239(
240 let
241 python =
242 let
243 packageOverrides = self: super: {
244 pandas = super.pandas.overridePythonAttrs (old: rec {
245 version = "0.19.1";
246 src = fetchPypi {
247 pname = "pandas";
248 inherit version;
249 hash = "sha256-JQn+rtpy/OA2deLszSKEuxyttqBzcAil50H+JDHUdCE=";
250 };
251 });
252 };
253 in
254 pkgs.python3.override {
255 inherit packageOverrides;
256 self = python;
257 };
258
259 in
260 python.withPackages (ps: [ ps.blaze ])
261).env
262```
263
264The next example shows a non trivial overriding of the `blas` implementation to
265be used through out all of the Python package set:
266
267```nix
268{
269 python3MyBlas = pkgs.python3.override {
270 packageOverrides = self: super: {
271 # We need toPythonModule for the package set to evaluate this
272 blas = super.toPythonModule (super.pkgs.blas.override { blasProvider = super.pkgs.mkl; });
273 lapack = super.toPythonModule (super.pkgs.lapack.override { lapackProvider = super.pkgs.mkl; });
274 };
275 };
276}
277```
278
279This is particularly useful for numpy and scipy users who want to gain speed with other blas implementations.
280Note that using `scipy = super.scipy.override { blas = super.pkgs.mkl; };` will likely result in
281compilation issues, because scipy dependencies need to use the same blas implementation as well.
282
283#### `buildPythonApplication` function {#buildpythonapplication-function}
284
285The [`buildPythonApplication`](#buildpythonapplication-function) function is practically the same as
286[`buildPythonPackage`](#buildpythonpackage-function). The main purpose of this function is to build a Python
287package where one is interested only in the executables, and not importable
288modules. For that reason, when adding this package to a [`python.buildEnv`](#python.buildenv-function), the
289modules won't be made available.
290
291Another difference is that [`buildPythonPackage`](#buildpythonpackage-function) by default prefixes the names of
292the packages with the version of the interpreter. Because this is irrelevant for
293applications, the prefix is omitted.
294
295When packaging a Python application with [`buildPythonApplication`](#buildpythonapplication-function), it should be
296called with `callPackage` and passed `python3` or `python3Packages` (possibly
297specifying an interpreter version), like this:
298
299```nix
300{
301 lib,
302 python3Packages,
303 fetchPypi,
304}:
305
306python3Packages.buildPythonApplication rec {
307 pname = "luigi";
308 version = "2.7.9";
309 pyproject = true;
310
311 src = fetchPypi {
312 inherit pname version;
313 hash = "sha256-Pe229rT0aHwA98s+nTHQMEFKZPo/yw6sot8MivFDvAw=";
314 };
315
316 build-system = with python3Packages; [ setuptools ];
317
318 dependencies = with python3Packages; [
319 tornado
320 python-daemon
321 ];
322
323 meta = {
324 # ...
325 };
326}
327```
328
329This is then added to `pkgs/by-name` just as any other application would be.
330
331Since the package is an application, a consumer doesn't need to care about
332Python versions or modules, which is why they don't go in `python3Packages`.
333
334#### `toPythonApplication` function {#topythonapplication-function}
335
336A distinction is made between applications and libraries, however, sometimes a
337package is used as both. In this case the package is added as a library to
338`python-packages.nix` and as an application to `pkgs/by-name`. To reduce
339duplication the `toPythonApplication` can be used to convert a library to an
340application.
341
342The Nix expression shall use [`buildPythonPackage`](#buildpythonpackage-function) and be called from
343`python-packages.nix`. A reference shall be created from `pkgs/by-name` to
344the attribute in `python-packages.nix`, and the `toPythonApplication` shall be
345applied to the reference:
346
347```nix
348{ python3Packages }:
349
350python3Packages.toPythonApplication python3Packages.youtube-dl
351```
352
353#### `toPythonModule` function {#topythonmodule-function}
354
355In some cases, such as bindings, a package is created using
356[`stdenv.mkDerivation`](#sec-using-stdenv) and added as attribute in `pkgs/by-name` or in `all-packages.nix`. The Python
357bindings should be made available from `python-packages.nix`. The
358`toPythonModule` function takes a derivation and makes certain Python-specific
359modifications.
360
361```nix
362{
363 opencv = toPythonModule (
364 pkgs.opencv.override {
365 enablePython = true;
366 pythonPackages = self;
367 }
368 );
369}
370```
371
372Do pay attention to passing in the right Python version!
373
374#### `mkPythonMetaPackage` function {#mkpythonmetapackage-function}
375
376This will create a meta package containing [metadata files](https://packaging.python.org/en/latest/specifications/recording-installed-packages/) to satisfy a dependency on a package, without it actually having been installed into the environment.
377In nixpkgs this is used to package Python packages with split binary/source distributions such as [psycopg2](https://pypi.org/project/psycopg2/)/[psycopg2-binary](https://pypi.org/project/psycopg2-binary/).
378
379```nix
380mkPythonMetaPackage {
381 pname = "psycopg2-binary";
382 inherit (psycopg2) optional-dependencies version;
383 dependencies = [ psycopg2 ];
384 meta = { inherit (psycopg2.meta) description homepage; };
385}
386```
387
388#### `mkPythonEditablePackage` function {#mkpythoneditablepackage-function}
389
390When developing Python packages it's common to install packages in [editable mode](https://setuptools.pypa.io/en/latest/userguide/development_mode.html).
391Like `mkPythonMetaPackage` this function exists to create an otherwise empty package, but also containing a pointer to an impure location outside the Nix store that can be changed without rebuilding.
392
393The editable root is passed as a string. Normally `.pth` files contains absolute paths to the mutable location. This isn't always ergonomic with Nix, so environment variables are expanded at runtime.
394This means that a shell hook setting up something like a `$REPO_ROOT` variable can be used as the relative package root.
395
396As an implementation detail, the [PEP-518](https://peps.python.org/pep-0518/) `build-system` specified won't be used, but instead the editable package will be built using [hatchling](https://pypi.org/project/hatchling/).
397The `build-system`'s provided will instead become runtime dependencies of the editable package.
398
399Note that overriding packages deeper in the dependency graph _can_ work, but it's not the primary use case and overriding existing packages can make others break in unexpected ways.
400
401```nix
402{
403 pkgs ? import <nixpkgs> { },
404}:
405
406let
407 pyproject = pkgs.lib.importTOML ./pyproject.toml;
408
409 myPython = pkgs.python.override {
410 self = myPython;
411 packageOverrides = pyfinal: pyprev: {
412 # An editable package with a script that loads our mutable location
413 my-editable = pyfinal.mkPythonEditablePackage {
414 # Inherit project metadata from pyproject.toml
415 pname = pyproject.project.name;
416 inherit (pyproject.project) version;
417
418 # The editable root passed as a string
419 root = "$REPO_ROOT/src"; # Use environment variable expansion at runtime
420
421 # Inject a script (other PEP-621 entrypoints are also accepted)
422 inherit (pyproject.project) scripts;
423 };
424 };
425 };
426
427 pythonEnv = myPython.withPackages (ps: [ ps.my-editable ]);
428
429in
430pkgs.mkShell { packages = [ pythonEnv ]; }
431```
432
433#### `python.buildEnv` function {#python.buildenv-function}
434
435Python environments can be created using the low-level `pkgs.buildEnv` function.
436This example shows how to create an environment that has the Pyramid Web Framework.
437Saving the following as `default.nix`
438
439```nix
440with import <nixpkgs> { };
441
442python3.buildEnv.override {
443 extraLibs = [ python3Packages.pyramid ];
444 ignoreCollisions = true;
445}
446```
447
448and running `nix-build` will create
449
450```
451/nix/store/cf1xhjwzmdki7fasgr4kz6di72ykicl5-python-2.7.8-env
452```
453
454with wrapped binaries in `bin/`.
455
456You can also use the `env` attribute to create local environments with needed
457packages installed. This is somewhat comparable to `virtualenv`. For example,
458running `nix-shell` with the following `shell.nix`
459
460```nix
461with import <nixpkgs> { };
462
463(python3.buildEnv.override {
464 extraLibs = with python3Packages; [
465 numpy
466 requests
467 ];
468}).env
469```
470
471will drop you into a shell where Python will have the
472specified packages in its path.
473
474##### `python.buildEnv` arguments {#python.buildenv-arguments}
475
476
477* `extraLibs`: List of packages installed inside the environment.
478* `postBuild`: Shell command executed after the build of environment.
479* `ignoreCollisions`: Ignore file collisions inside the environment (default is `false`).
480* `permitUserSite`: Skip setting the `PYTHONNOUSERSITE` environment variable in
481 wrapped binaries in the environment.
482
483#### `python.withPackages` function {#python.withpackages-function}
484
485The [`python.withPackages`](#python.withpackages-function) function provides a simpler interface to the [`python.buildEnv`](#python.buildenv-function) functionality.
486It takes a function as an argument that is passed the set of python packages and returns the list
487of the packages to be included in the environment. Using the [`withPackages`](#python.withpackages-function) function, the previous
488example for the Pyramid Web Framework environment can be written like this:
489
490```nix
491with import <nixpkgs> { };
492
493python.withPackages (ps: [ ps.pyramid ])
494```
495
496[`withPackages`](#python.withpackages-function) passes the correct package set for the specific interpreter
497version as an argument to the function. In the above example, `ps` equals
498`pythonPackages`. But you can also easily switch to using python3:
499
500```nix
501with import <nixpkgs> { };
502
503python3.withPackages (ps: [ ps.pyramid ])
504```
505
506Now, `ps` is set to `python3Packages`, matching the version of the interpreter.
507
508As [`python.withPackages`](#python.withpackages-function) uses [`python.buildEnv`](#python.buildenv-function) under the hood, it also
509supports the `env` attribute. The `shell.nix` file from the previous section can
510thus be also written like this:
511
512```nix
513with import <nixpkgs> { };
514
515(python3.withPackages (
516 ps: with ps; [
517 numpy
518 requests
519 ]
520)).env
521```
522
523In contrast to [`python.buildEnv`](#python.buildenv-function), [`python.withPackages`](#python.withpackages-function) does not support the
524more advanced options such as `ignoreCollisions = true` or `postBuild`. If you
525need them, you have to use [`python.buildEnv`](#python.buildenv-function).
526
527Python 2 namespace packages may provide `__init__.py` that collide. In that case
528[`python.buildEnv`](#python.buildenv-function) should be used with `ignoreCollisions = true`.
529
530#### Setup hooks {#setup-hooks}
531
532The following are setup hooks specifically for Python packages. Most of these
533are used in [`buildPythonPackage`](#buildpythonpackage-function).
534
535- `eggUnpackhook` to move an egg to the correct folder so it can be installed
536 with the `eggInstallHook`
537- `eggBuildHook` to skip building for eggs.
538- `eggInstallHook` to install eggs.
539- `pypaBuildHook` to build a wheel using
540 [`pypa/build`](https://pypa-build.readthedocs.io/en/latest/index.html) and
541 PEP 517/518. Note a build system (e.g. `setuptools` or `flit`) should still
542 be added as `build-system`.
543- `pypaInstallHook` to install wheels.
544- `pytestCheckHook` to run tests with `pytest`. See [example usage](#using-pytestcheckhook).
545- `pythonCatchConflictsHook` to fail if the package depends on two different versions of the same dependency.
546- `pythonImportsCheckHook` to check whether importing the listed modules works.
547- `pythonRelaxDepsHook` will relax Python dependencies restrictions for the package.
548 See [example usage](#using-pythonrelaxdepshook).
549- `pythonRemoveBinBytecode` to remove bytecode from the `/bin` folder.
550- `setuptoolsBuildHook` to build a wheel using `setuptools`.
551- `sphinxHook` to build documentation and manpages using Sphinx.
552- `venvShellHook` to source a Python 3 `venv` at the `venvDir` location. A
553 `venv` is created if it does not yet exist. `postVenvCreation` can be used to
554 to run commands only after venv is first created.
555- `wheelUnpackHook` to move a wheel to the correct folder so it can be installed
556 with the `pipInstallHook`.
557- `unittestCheckHook` will run tests with `python -m unittest discover`. See [example usage](#using-unittestcheckhook).
558
559## User Guide {#user-guide}
560
561### Using Python {#using-python}
562
563#### Overview {#overview}
564
565Several versions of the Python interpreter are available on Nix, as well as a
566high amount of packages. The attribute `python3` refers to the default
567interpreter, which is currently CPython 3.13. The attribute `python` refers to
568CPython 2.7 for backwards compatibility. It is also possible to refer to
569specific versions, e.g., `python313` refers to CPython 3.13, and `pypy` refers to
570the default PyPy interpreter.
571
572Python is used a lot, and in different ways. This affects also how it is
573packaged. In the case of Python on Nix, an important distinction is made between
574whether the package is considered primarily an application, or whether it should
575be used as a library, i.e., of primary interest are the modules in
576`site-packages` that should be importable.
577
578In the Nixpkgs tree Python applications can be found throughout, depending on
579what they do, and are called from the main package set. Python libraries,
580however, are in separate sets, with one set per interpreter version.
581
582The interpreters have several common attributes. One of these attributes is
583`pkgs`, which is a package set of Python libraries for this specific
584interpreter. E.g., the `toolz` package corresponding to the default interpreter
585is `python3.pkgs.toolz`, and the CPython 3.13 version is `python313.pkgs.toolz`.
586The main package set contains aliases to these package sets, e.g.
587`pythonPackages` refers to `python.pkgs` and `python313Packages` to
588`python313.pkgs`.
589
590#### Installing Python and packages {#installing-python-and-packages}
591
592The Nix and NixOS manuals explain how packages are generally installed. In the
593case of Python and Nix, it is important to make a distinction between whether the
594package is considered an application or a library.
595
596Applications on Nix are typically installed into your user profile imperatively
597using `nix-env -i`, and on NixOS declaratively by adding the package name to
598`environment.systemPackages` in `/etc/nixos/configuration.nix`. Dependencies
599such as libraries are automatically installed and should not be installed
600explicitly.
601
602The same goes for Python applications. Python applications can be installed in
603your profile, and will be wrapped to find their exact library dependencies,
604without impacting other applications or polluting your user environment.
605
606But Python libraries you would like to use for development cannot be installed,
607at least not individually, because they won't be able to find each other
608resulting in import errors. Instead, it is possible to create an environment
609with [`python.buildEnv`](#python.buildenv-function) or [`python.withPackages`](#python.withpackages-function) where the interpreter and other
610executables are wrapped to be able to find each other and all of the modules.
611
612In the following examples we will start by creating a simple, ad-hoc environment
613with a nix-shell that has `numpy` and `toolz` in Python 3.13; then we will create
614a re-usable environment in a single-file Python script; then we will create a
615full Python environment for development with this same environment.
616
617Philosophically, this should be familiar to users who are used to a `venv` style
618of development: individual projects create their own Python environments without
619impacting the global environment or each other.
620
621#### Ad-hoc temporary Python environment with `nix-shell` {#ad-hoc-temporary-python-environment-with-nix-shell}
622
623The simplest way to start playing with the way nix wraps and sets up Python
624environments is with `nix-shell` at the cmdline. These environments create a
625temporary shell session with a Python and a *precise* list of packages (plus
626their runtime dependencies), with no other Python packages in the Python
627interpreter's scope.
628
629To create a Python 3.13 session with `numpy` and `toolz` available, run:
630
631```sh
632$ nix-shell -p 'python313.withPackages(ps: with ps; [ numpy toolz ])'
633```
634
635By default `nix-shell` will start a `bash` session with this interpreter in our
636`PATH`, so if we then run:
637
638```Python console
639[nix-shell:~/src/nixpkgs]$ python3
640Python 3.13.3 (main, Apr 8 2025, 13:54:08) [GCC 14.2.1 20250322] on linux
641Type "help", "copyright", "credits" or "license" for more information.
642>>> import numpy; import toolz
643```
644
645Note that no other modules are in scope, even if they were imperatively
646installed into our user environment as a dependency of a Python application:
647
648```Python console
649>>> import requests
650Traceback (most recent call last):
651 File "<stdin>", line 1, in <module>
652ModuleNotFoundError: No module named 'requests'
653```
654
655We can add as many additional modules onto the `nix-shell` as we need, and we
656will still get 1 wrapped Python interpreter. We can start the interpreter
657directly like so:
658
659```sh
660$ nix-shell -p "python313.withPackages (ps: with ps; [ numpy toolz requests ])" --run python3
661Python 3.13.3 (main, Apr 8 2025, 13:54:08) [GCC 14.2.1 20250322] on linux
662Type "help", "copyright", "credits" or "license" for more information.
663>>> import requests
664>>>
665```
666
667Notice that this time it built a new Python environment, which now includes
668`requests`. Building an environment just creates wrapper scripts that expose the
669selected dependencies to the interpreter while re-using the actual modules. This
670means if any other env has installed `requests` or `numpy` in a different
671context, we don't need to recompile them -- we just recompile the wrapper script
672that sets up an interpreter pointing to them. This matters much more for "big"
673modules like `pytorch` or `tensorflow`.
674
675Module names usually match their names on [pypi.org](https://pypi.org/), but
676normalized according to PEP 503/508. (e.g. Foo__Bar.baz -> foo-bar-baz)
677You can use the [Nixpkgs search website](https://nixos.org/nixos/packages.html)
678to find them as well (along with non-python packages).
679
680At this point we can create throwaway experimental Python environments with
681arbitrary dependencies. This is a good way to get a feel for how the Python
682interpreter and dependencies work in Nix and NixOS, but to do some actual
683development, we'll want to make it a bit more persistent.
684
685##### Running Python scripts and using `nix-shell` as shebang {#running-python-scripts-and-using-nix-shell-as-shebang}
686
687Sometimes, we have a script whose header looks like this:
688
689```python
690#!/usr/bin/env python3
691import numpy as np
692a = np.array([1,2])
693b = np.array([3,4])
694print(f"The dot product of {a} and {b} is: {np.dot(a, b)}")
695```
696
697Executing this script requires a `python3` that has `numpy`. Using what we learned
698in the previous section, we could startup a shell and just run it like so:
699
700```ShellSession
701$ nix-shell -p 'python313.withPackages (ps: with ps; [ numpy ])' --run 'python3 foo.py'
702The dot product of [1 2] and [3 4] is: 11
703```
704
705But if we maintain the script ourselves, and if there are more dependencies, it
706may be nice to encode those dependencies in source to make the script re-usable
707without that bit of knowledge. That can be done by using `nix-shell` as a
708[shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)), like so:
709
710```python
711#!/usr/bin/env nix-shell
712#!nix-shell -i python3 -p "python3.withPackages(ps: [ ps.numpy ])"
713import numpy as np
714a = np.array([1,2])
715b = np.array([3,4])
716print(f"The dot product of {a} and {b} is: {np.dot(a, b)}")
717```
718
719Then we execute it, without requiring any environment setup at all!
720
721```sh
722$ ./foo.py
723The dot product of [1 2] and [3 4] is: 11
724```
725
726If the dependencies are not available on the host where `foo.py` is executed, it
727will build or download them from a Nix binary cache prior to starting up, prior
728that it is executed on a machine with a multi-user Nix installation.
729
730This provides a way to ship a self bootstrapping Python script, akin to a
731statically linked binary, where it can be run on any machine (provided nix is
732installed) without having to assume that `numpy` is installed globally on the
733system.
734
735By default it is pulling the import checkout of Nixpkgs itself from our nix
736channel, which is nice as it cache-aligns with our other package builds, but we
737can make it fully reproducible by pinning the `nixpkgs` import:
738
739```python
740#!/usr/bin/env nix-shell
741#!nix-shell -i python3 -p "python3.withPackages (ps: [ ps.numpy ])"
742#!nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/e51209796c4262bfb8908e3d6d72302fe4e96f5f.tar.gz
743import numpy as np
744a = np.array([1,2])
745b = np.array([3,4])
746print(f"The dot product of {a} and {b} is: {np.dot(a, b)}")
747```
748
749This will execute with the exact same versions of Python 3.10, numpy, and system
750dependencies a year from now as it does today, because it will always use
751exactly git commit `e51209796c4262bfb8908e3d6d72302fe4e96f5f` of Nixpkgs for all
752of the package versions.
753
754This is also a great way to ensure the script executes identically on different
755servers.
756
757##### Load environment from `.nix` expression {#load-environment-from-.nix-expression}
758
759We've now seen how to create an ad-hoc temporary shell session, and how to
760create a single script with Python dependencies, but in the course of normal
761development we're usually working in an entire package repository.
762
763As explained [in the `nix-shell` section](https://nixos.org/manual/nix/stable/command-ref/nix-shell) of the Nix manual, `nix-shell` can also load an expression from a `.nix` file.
764Say we want to have Python 3.13, `numpy` and `toolz`, like before,
765in an environment. We can add a `shell.nix` file describing our dependencies:
766
767```nix
768with import <nixpkgs> { };
769(python313.withPackages (
770 ps: with ps; [
771 numpy
772 toolz
773 ]
774)).env
775```
776
777And then at the command line, just typing `nix-shell` produces the same
778environment as before. In a normal project, we'll likely have many more
779dependencies; this can provide a way for developers to share the environments
780with each other and with CI builders.
781
782What's happening here?
783
7841. We begin with importing the Nix Packages collections. `import <nixpkgs>`
785 imports the `<nixpkgs>` function, `{}` calls it and the `with` statement
786 brings all attributes of `nixpkgs` in the local scope. These attributes form
787 the main package set.
7882. Then we create a Python 3.13 environment with the [`withPackages`](#python.withpackages-function) function, as before.
7893. The [`withPackages`](#python.withpackages-function) function expects us to provide a function as an argument
790 that takes the set of all Python packages and returns a list of packages to
791 include in the environment. Here, we select the packages `numpy` and `toolz`
792 from the package set.
793
794To combine this with `mkShell` you can:
795
796```nix
797with import <nixpkgs> { };
798let
799 pythonEnv = python313.withPackages (ps: [
800 ps.numpy
801 ps.toolz
802 ]);
803in
804mkShell {
805 packages = [
806 pythonEnv
807
808 black
809 mypy
810
811 libffi
812 openssl
813 ];
814}
815```
816
817This will create a unified environment that has not just our Python interpreter
818and its Python dependencies, but also tools like `black` or `mypy` and libraries
819like `libffi` and `openssl` in scope. This is generic and can span any number of
820tools or languages across the Nixpkgs ecosystem.
821
822##### Installing environments globally on the system {#installing-environments-globally-on-the-system}
823
824Up to now, we've been creating environments scoped to an ad-hoc shell session,
825or a single script, or a single project. This is generally advisable, as it
826avoids pollution across contexts.
827
828However, sometimes we know we will often want a Python with some basic packages,
829and want this available without having to enter into a shell or build context.
830This can be useful to have things like vim/emacs editors and plugins or shell
831tools "just work" without having to set them up, or when running other software
832that expects packages to be installed globally.
833
834To create your own custom environment, create a file in `~/.config/nixpkgs/overlays/`
835that looks like this:
836
837```nix
838# ~/.config/nixpkgs/overlays/myEnv.nix
839self: super: {
840 myEnv = super.buildEnv {
841 name = "myEnv";
842 paths = [
843 # A Python 3 interpreter with some packages
844 (self.python3.withPackages (
845 ps: with ps; [
846 pyflakes
847 pytest
848 black
849 ]
850 ))
851
852 # Some other packages we'd like as part of this env
853 self.mypy
854 self.black
855 self.ripgrep
856 self.tmux
857 ];
858 };
859}
860```
861
862You can then build and install this to your profile with:
863
864```sh
865nix-env -iA myEnv
866```
867
868One limitation of this is that you can only have 1 Python env installed
869globally, since they conflict on the `python` to load out of your `PATH`.
870
871If you get a conflict or prefer to keep the setup clean, you can have `nix-env`
872atomically *uninstall* all other imperatively installed packages and replace
873your profile with just `myEnv` by using the `--replace` flag.
874
875##### Environment defined in `/etc/nixos/configuration.nix` {#environment-defined-in-etcnixosconfiguration.nix}
876
877For the sake of completeness, here's how to install the environment system-wide
878on NixOS.
879
880```nix
881{
882 # ...
883
884 environment.systemPackages = with pkgs; [
885 (python310.withPackages (
886 ps: with ps; [
887 numpy
888 toolz
889 ]
890 ))
891 ];
892}
893```
894
895### Developing with Python {#developing-with-python}
896
897Above, we were mostly just focused on use cases and what to do to get started
898creating working Python environments in nix.
899
900Now that you know the basics to be up and running, it is time to take a step
901back and take a deeper look at how Python packages are packaged on Nix.
902
903#### Python library packages in Nixpkgs {#python-library-packages-in-nixpkgs}
904
905With Nix all packages are built by functions. The main function in Nix for
906building Python libraries is [`buildPythonPackage`](#buildpythonpackage-function). Let's see how we can build the
907`toolz` package.
908
909```nix
910{
911 lib,
912 buildPythonPackage,
913 fetchPypi,
914 setuptools,
915}:
916
917buildPythonPackage rec {
918 pname = "toolz";
919 version = "0.10.0";
920 pyproject = true;
921
922 src = fetchPypi {
923 inherit pname version;
924 hash = "sha256-CP3V73yWSArRHBLUct4hrNMjWZlvaaUlkpm1QP66RWA=";
925 };
926
927 build-system = [ setuptools ];
928
929 # has no tests
930 doCheck = false;
931
932 pythonImportsCheck = [
933 "toolz.itertoolz"
934 "toolz.functoolz"
935 "toolz.dicttoolz"
936 ];
937
938 meta = {
939 changelog = "https://github.com/pytoolz/toolz/releases/tag/${version}";
940 homepage = "https://github.com/pytoolz/toolz";
941 description = "List processing tools and functional utilities";
942 license = lib.licenses.bsd3;
943 };
944}
945```
946
947What happens here? The function [`buildPythonPackage`](#buildpythonpackage-function) is called and as argument
948it accepts a set. In this case the set is a recursive set, `rec`. One of the
949arguments is the name of the package, which consists of a basename (generally
950following the name on PyPI) and a version. Another argument, `src` specifies the
951source, which in this case is fetched from PyPI using the helper function
952`fetchPypi`. The argument `doCheck` is used to set whether tests should be run
953when building the package. Since there are no tests, we rely on [`pythonImportsCheck`](#using-pythonimportscheck)
954to test whether the package can be imported. Furthermore, we specify some meta
955information. The output of the function is a derivation.
956
957An expression for `toolz` can be found in the Nixpkgs repository. As explained
958in the introduction of this Python section, a derivation of `toolz` is available
959for each interpreter version, e.g. `python313.pkgs.toolz` refers to the `toolz`
960derivation corresponding to the CPython 3.13 interpreter.
961
962The above example works when you're directly working on
963`pkgs/top-level/python-packages.nix` in the Nixpkgs repository. Often though,
964you will want to test a Nix expression outside of the Nixpkgs tree.
965
966The following expression creates a derivation for the `toolz` package,
967and adds it along with a `numpy` package to a Python environment.
968
969```nix
970with import <nixpkgs> { };
971
972(
973 let
974 my_toolz = python313.pkgs.buildPythonPackage rec {
975 pname = "toolz";
976 version = "0.10.0";
977 pyproject = true;
978
979 src = fetchPypi {
980 inherit pname version;
981 hash = "sha256-CP3V73yWSArRHBLUct4hrNMjWZlvaaUlkpm1QP66RWA=";
982 };
983
984 build-system = [ python313.pkgs.setuptools ];
985
986 # has no tests
987 doCheck = false;
988
989 meta = {
990 homepage = "https://github.com/pytoolz/toolz/";
991 description = "List processing tools and functional utilities";
992 # [...]
993 };
994 };
995
996 in
997 python313.withPackages (
998 ps: with ps; [
999 numpy
1000 my_toolz
1001 ]
1002 )
1003).env
1004```
1005
1006Executing `nix-shell` will result in an environment in which you can use
1007Python 3.13 and the `toolz` package. As you can see we had to explicitly mention
1008for which Python version we want to build a package.
1009
1010So, what did we do here? Well, we took the Nix expression that we used earlier
1011to build a Python environment, and said that we wanted to include our own
1012version of `toolz`, named `my_toolz`. To introduce our own package in the scope
1013of [`withPackages`](#python.withpackages-function) we used a `let` expression. You can see that we used
1014`ps.numpy` to select numpy from the nixpkgs package set (`ps`). We did not take
1015`toolz` from the Nixpkgs package set this time, but instead took our own version
1016that we introduced with the `let` expression.
1017
1018#### Handling dependencies {#handling-dependencies}
1019
1020Our example, `toolz`, does not have any dependencies on other Python packages or system libraries.
1021[`buildPythonPackage`](#buildpythonpackage-function) uses the the following arguments in the following circumstances:
1022
1023- `dependencies` - For Python runtime dependencies.
1024- `build-system` - For Python build-time requirements.
1025- [`buildInputs`](#var-stdenv-buildInputs) - For non-Python build-time requirements.
1026- [`nativeCheckInputs`](#var-stdenv-nativeCheckInputs) - For test dependencies
1027
1028Dependencies can belong to multiple arguments, for example if something is both a build time requirement & a runtime dependency.
1029
1030The following example shows which arguments are given to [`buildPythonPackage`](#buildpythonpackage-function) in
1031order to build [`datashape`](https://github.com/blaze/datashape).
1032
1033```nix
1034{
1035 lib,
1036 buildPythonPackage,
1037 fetchPypi,
1038
1039 # build dependencies
1040 setuptools,
1041
1042 # dependencies
1043 numpy,
1044 multipledispatch,
1045 python-dateutil,
1046
1047 # tests
1048 pytestCheckHook,
1049}:
1050
1051buildPythonPackage rec {
1052 pname = "datashape";
1053 version = "0.4.7";
1054 pyproject = true;
1055
1056 src = fetchPypi {
1057 inherit pname version;
1058 hash = "sha256-FLLvdm1MllKrgTGC6Gb0k0deZeVYvtCCLji/B7uhong=";
1059 };
1060
1061 build-system = [ setuptools ];
1062
1063 dependencies = [
1064 multipledispatch
1065 numpy
1066 python-dateutil
1067 ];
1068
1069 nativeCheckInputs = [ pytestCheckHook ];
1070
1071 meta = {
1072 changelog = "https://github.com/blaze/datashape/releases/tag/${version}";
1073 homepage = "https://github.com/ContinuumIO/datashape";
1074 description = "Data description language";
1075 license = lib.licenses.bsd2;
1076 };
1077}
1078```
1079
1080We can see several runtime dependencies, `numpy`, `multipledispatch`, and
1081`python-dateutil`. Furthermore, we have [`nativeCheckInputs`](#var-stdenv-nativeCheckInputs) with `pytestCheckHook`.
1082`pytestCheckHook` is a test runner hook and is only used during the [`checkPhase`](#ssec-check-phase) and is
1083therefore not added to `dependencies`.
1084
1085In the previous case we had only dependencies on other Python packages to consider.
1086Occasionally you have also system libraries to consider. E.g., `lxml` provides
1087Python bindings to `libxml2` and `libxslt`. These libraries are only required
1088when building the bindings and are therefore added as [`buildInputs`](#var-stdenv-buildInputs).
1089
1090```nix
1091{
1092 lib,
1093 buildPythonPackage,
1094 fetchPypi,
1095 setuptools,
1096 libxml2,
1097 libxslt,
1098}:
1099
1100buildPythonPackage rec {
1101 pname = "lxml";
1102 version = "3.4.4";
1103 pyproject = true;
1104
1105 src = fetchPypi {
1106 inherit pname version;
1107 hash = "sha256-s9NiusRxFydHzaNRMjjxFcvWxfi45jGb9ql6eJJyQJk=";
1108 };
1109
1110 build-system = [ setuptools ];
1111
1112 buildInputs = [
1113 libxml2
1114 libxslt
1115 ];
1116
1117 # tests are meant to be ran "in-place" in the same directory as src
1118 doCheck = false;
1119
1120 pythonImportsCheck = [
1121 "lxml"
1122 "lxml.etree"
1123 ];
1124
1125 meta = {
1126 changelog = "https://github.com/lxml/lxml/releases/tag/lxml-${version}";
1127 description = "Pythonic binding for the libxml2 and libxslt libraries";
1128 homepage = "https://lxml.de";
1129 license = lib.licenses.bsd3;
1130 maintainers = with lib.maintainers; [ sjourdois ];
1131 };
1132}
1133```
1134
1135In this example `lxml` and Nix are able to work out exactly where the relevant
1136files of the dependencies are. This is not always the case.
1137
1138The example below shows bindings to The Fastest Fourier Transform in the West,
1139commonly known as FFTW. On Nix we have separate packages of FFTW for the
1140different types of floats (`"single"`, `"double"`, `"long-double"`). The
1141bindings need all three types, and therefore we add all three as [`buildInputs`](#var-stdenv-buildInputs).
1142The bindings don't expect to find each of them in a different folder, and
1143therefore we have to set `LDFLAGS` and `CFLAGS`.
1144
1145```nix
1146{
1147 lib,
1148 buildPythonPackage,
1149 fetchPypi,
1150
1151 # build dependencies
1152 setuptools,
1153
1154 # dependencies
1155 fftw,
1156 fftwFloat,
1157 fftwLongDouble,
1158 numpy,
1159 scipy,
1160}:
1161
1162buildPythonPackage rec {
1163 pname = "pyfftw";
1164 version = "0.9.2";
1165 pyproject = true;
1166
1167 src = fetchPypi {
1168 inherit pname version;
1169 hash = "sha256-9ru2r6kwhUCaskiFoaPNuJCfCVoUL01J40byvRt4kHQ=";
1170 };
1171
1172 build-system = [ setuptools ];
1173
1174 buildInputs = [
1175 fftw
1176 fftwFloat
1177 fftwLongDouble
1178 ];
1179
1180 dependencies = [
1181 numpy
1182 scipy
1183 ];
1184
1185 preConfigure = ''
1186 export LDFLAGS="-L${fftw.dev}/lib -L${fftwFloat.out}/lib -L${fftwLongDouble.out}/lib"
1187 export CFLAGS="-I${fftw.dev}/include -I${fftwFloat.dev}/include -I${fftwLongDouble.dev}/include"
1188 '';
1189
1190 # Tests cannot import pyfftw. pyfftw works fine though.
1191 doCheck = false;
1192
1193 pythonImportsCheck = [ "pyfftw" ];
1194
1195 meta = {
1196 changelog = "https://github.com/pyFFTW/pyFFTW/releases/tag/v${version}";
1197 description = "Pythonic wrapper around FFTW, the FFT library, presenting a unified interface for all the supported transforms";
1198 homepage = "http://hgomersall.github.com/pyFFTW";
1199 license = with lib.licenses; [
1200 bsd2
1201 bsd3
1202 ];
1203 };
1204}
1205```
1206
1207Note also the line [`doCheck = false;`](#var-stdenv-doCheck), we explicitly disabled running the test-suite.
1208
1209#### Testing Python Packages {#testing-python-packages}
1210
1211It is highly encouraged to have testing as part of the package build. This
1212helps to avoid situations where the package was able to build and install,
1213but is not usable at runtime.
1214Your package should provide its own [`checkPhase`](#ssec-check-phase).
1215
1216::: {.note}
1217The [`checkPhase`](#ssec-check-phase) for python maps to the `installCheckPhase` on a
1218normal derivation. This is due to many python packages not behaving well
1219to the pre-installed version of the package. Version info, and natively
1220compiled extensions generally only exist in the install directory, and
1221thus can cause issues when a test suite asserts on that behavior.
1222:::
1223
1224::: {.note}
1225Tests should only be disabled if they don't agree with nix
1226(e.g. external dependencies, network access, flakey tests), however,
1227as many tests should be enabled as possible. Failing tests can still be
1228a good indication that the package is not in a valid state.
1229:::
1230
1231::: {.note}
1232We only want to test the functionality of a package. In particular, we are not
1233interested in coverage, formatting, and type checking. If pytest fails with
1234`unrecognized arguments: --cov`, add `pytest-cov-stub` to `nativeCheckInputs`
1235rather than `pytest-cov`.
1236:::
1237
1238#### Using pytest {#using-pytest}
1239
1240Pytest is the most common test runner for python repositories. A trivial
1241test run would be:
1242
1243```nix
1244{
1245 nativeCheckInputs = [ pytest ];
1246 checkPhase = ''
1247 runHook preCheck
1248
1249 pytest
1250
1251 runHook postCheck
1252 '';
1253}
1254```
1255
1256However, many repositories' test suites do not translate well to nix's build
1257sandbox, and will generally need many tests to be disabled.
1258
1259This is achievable by
1260- Including paths or test items (`path/to/file.py::MyClass` or `path/to/file.py::MyClass::test_method`) with positional arguments.
1261- Excluding paths with `--ignore` or globbed paths with `--ignore-glob`.
1262- Excluding test items using the `--deselect` flag.
1263- Including or excluding classes or test methods by their name using the `-k` flag.
1264- Including or excluding test by their marks using the `-m` flag.
1265
1266We highly recommend `pytestCheckHook` for an easier and more structural setup.
1267
1268#### Using pytestCheckHook {#using-pytestcheckhook}
1269
1270`pytestCheckHook` is a convenient hook which will set up (or configure)
1271a [`checkPhase`](#ssec-check-phase) to run `pytest`. This is also beneficial
1272when a package may need many items disabled to run the test suite.
1273Most packages use `pytest` or `unittest`, which is compatible with `pytest`,
1274so you will most likely use `pytestCheckHook`.
1275
1276To use `pytestCheckHook`, add it to `nativeCheckInputs`.
1277Adding `pytest` is not required, since it is included with `pytestCheckHook`.
1278
1279```nix
1280{ nativeCheckInputs = [ pytestCheckHook ]; }
1281```
1282
1283`pytestCheckHook` recognizes the following attributes:
1284
1285`enabledTestPaths` and `disabledTestPaths`
1286
1287: To specify path globs (files or directories) or test items.
1288
1289`enabledTests` and `disabledTests`
1290
1291: To specify keywords for class names or test method names.
1292
1293`enabledTestMarks` and `disabledTestMarks`
1294
1295: To specify test marks.
1296
1297`pytestFlags`
1298
1299: To append additional command-line arguments to `pytest`.
1300
1301By default, `pytest` automatically discovers which tests to run.
1302If tests are explicitly enabled, only those tests will run.
1303A test, that is both enabled and disabled, will not run.
1304
1305The following example demonstrates usage of various `pytestCheckHook` attributes:
1306
1307```nix
1308{
1309 nativeCheckInputs = [ pytestCheckHook ];
1310
1311 # Allow running the following test paths and test objects.
1312 enabledTestPaths = [
1313 # Find tests under the tests directory.
1314 # The trailing slash is not necessary.
1315 "tests/"
1316
1317 # Additionally run test_foo
1318 "other-tests/test_foo.py::Foo::test_foo"
1319 ];
1320
1321 # Override the above-enabled test paths and test objects.
1322 disabledTestPaths = [
1323 # Tests under tests/integration requires additional data.
1324 "tests/integration"
1325 ];
1326
1327 # Allow tests by keywords matching their class names or method names.
1328 enabledTests = [
1329 # pytest by default only runs test methods begin with "test_" or end with "_test".
1330 # This includes all functions whose name contains "test".
1331 "test"
1332 ];
1333
1334 # Override the above-enabled tests by keywords matching their class names or method names.
1335 disabledTests = [
1336 # Tests touching networks.
1337 "upload"
1338 "download"
1339 ];
1340
1341 # Additional pytest flags
1342 pytestFlags = [
1343 # Disable benchmarks and run benchmarking tests only once.
1344 "--benchmark-disable"
1345 ];
1346}
1347```
1348
1349These attributes are all passed into the derivation directly
1350and added to the `pytest` command without additional Bash expansion.
1351It requires `__structuredAttrs = true` to pass list elements containing spaces.
1352
1353The `<enabled/disabled>TestsPaths` attributes expand Unix-style globs.
1354If a test path contains characters like `*`, `?`, `[`, or `]`, you can
1355quote them with square brackets (`[*]`, `[?]`, `[[]`, and `[]]`) to match literally.
1356
1357The `<enabled/disabled>Tests` and `<enabled/disabled>TestMarks` attribute pairs
1358form a logical expression `((included_element1) or (included_element2)) and not (excluded_element1) and not (excluded_element2)`
1359which will be passed to pytest's `-k` and `-m` flags respectively.
1360With `__structuredAttrs = true` enabled, they additionally support sub-expressions.
1361
1362For example, you could disable test items like `TestFoo::test_bar_functionality`
1363by disabling tests that match both `"Foo"` **and** `"bar"`:
1364
1365```nix
1366{
1367 __structuredAttrs = true;
1368
1369 disabledTests = [ "Foo and bar" ];
1370}
1371```
1372
1373The main benefits of using `pytestCheckHook` to construct `pytest` commands
1374is structuralization and eval-time accessibility.
1375This is especially helpful to select tests or specify flags conditionally:
1376
1377```nix
1378{
1379 disabledTests = [
1380 # touches network
1381 "download"
1382 "update"
1383 ]
1384 ++ lib.optionals (pythonAtLeast "3.8") [
1385 # broken due to python3.8 async changes
1386 "async"
1387 ]
1388 ++ lib.optionals stdenv.buildPlatform.isDarwin [
1389 # can fail when building with other packages
1390 "socket"
1391 ];
1392}
1393```
1394
1395#### Using pythonImportsCheck {#using-pythonimportscheck}
1396
1397Although unit tests are highly preferred to validate correctness of a package, not
1398all packages have test suites that can be run easily, and some have none at all.
1399To help ensure the package still works, [`pythonImportsCheck`](#using-pythonimportscheck) can attempt to import
1400the listed modules.
1401
1402```nix
1403{
1404 pythonImportsCheck = [
1405 "requests"
1406 "urllib"
1407 ];
1408}
1409```
1410
1411roughly translates to:
1412
1413```nix
1414{
1415 postCheck = ''
1416 PYTHONPATH=$out/${python.sitePackages}:$PYTHONPATH
1417 python -c "import requests; import urllib"
1418 '';
1419}
1420```
1421
1422However, this is done in its own phase, and not dependent on whether [`doCheck = true;`](#var-stdenv-doCheck).
1423
1424This can also be useful in verifying that the package doesn't assume commonly
1425present packages (e.g. `setuptools`).
1426
1427#### Using pythonRelaxDepsHook {#using-pythonrelaxdepshook}
1428
1429It is common for upstream to specify a range of versions for its package
1430dependencies. This makes sense, since it ensures that the package will be built
1431with a subset of packages that is well tested. However, this commonly causes
1432issues when packaging in Nixpkgs, because the dependencies that this package
1433may need are too new or old for the package to build correctly. We also cannot
1434package multiple versions of the same package since this may cause conflicts
1435in `PYTHONPATH`.
1436
1437One way to side step this issue is to relax the dependencies. This can be done
1438by either removing the package version range or by removing the package
1439declaration entirely. This can be done using the `pythonRelaxDepsHook` hook. For
1440example, given the following `requirements.txt` file:
1441
1442```
1443pkg1<1.0
1444pkg2
1445pkg3>=1.0,<=2.0
1446```
1447
1448we can do:
1449
1450```nix
1451{
1452 pythonRelaxDeps = [
1453 "pkg1"
1454 "pkg3"
1455 ];
1456 pythonRemoveDeps = [ "pkg2" ];
1457}
1458```
1459
1460which would result in the following `requirements.txt` file:
1461
1462```
1463pkg1
1464pkg3
1465```
1466
1467Another option is to pass `true`, that will relax/remove all dependencies, for
1468example:
1469
1470```nix
1471{ pythonRelaxDeps = true; }
1472```
1473
1474which would result in the following `requirements.txt` file:
1475
1476```
1477pkg1
1478pkg2
1479pkg3
1480```
1481
1482In general you should always use `pythonRelaxDeps`, because `pythonRemoveDeps`
1483will convert build errors into runtime errors. However `pythonRemoveDeps` may
1484still be useful in exceptional cases, and also to remove dependencies wrongly
1485declared by upstream (for example, declaring `black` as a runtime dependency
1486instead of a dev dependency).
1487
1488Keep in mind that while the examples above are done with `requirements.txt`,
1489`pythonRelaxDepsHook` works by modifying the resulting wheel file, so it should
1490work with any of the [existing hooks](#setup-hooks).
1491
1492The `pythonRelaxDepsHook` has no effect on build time dependencies, such as
1493those specified in `build-system`. If a package requires incompatible build
1494time dependencies, they should be removed in `postPatch` through
1495`substituteInPlace` or similar.
1496
1497For ease of use, both `buildPythonPackage` and `buildPythonApplication` will
1498automatically add `pythonRelaxDepsHook` if either `pythonRelaxDeps` or
1499`pythonRemoveDeps` is specified.
1500
1501#### Using unittestCheckHook {#using-unittestcheckhook}
1502
1503`unittestCheckHook` is a hook which will set up (or configure) a [`checkPhase`](#ssec-check-phase) to run `python -m unittest discover`:
1504
1505```nix
1506{
1507 nativeCheckInputs = [ unittestCheckHook ];
1508
1509 unittestFlags = [
1510 "-s"
1511 "tests"
1512 "-v"
1513 ];
1514}
1515```
1516
1517`pytest` is compatible with `unittest`, so in most cases you can use `pytestCheckHook` instead.
1518
1519#### Using sphinxHook {#using-sphinxhook}
1520
1521The `sphinxHook` is a helpful tool to build documentation and manpages
1522using the popular Sphinx documentation generator.
1523It is setup to automatically find common documentation source paths and
1524render them using the default `html` style.
1525
1526```nix
1527{
1528 outputs = [
1529 "out"
1530 "doc"
1531 ];
1532
1533 nativeBuildInputs = [ sphinxHook ];
1534}
1535```
1536
1537The hook will automatically build and install the artifact into the
1538`doc` output, if it exists. It also provides an automatic diversion
1539for the artifacts of the `man` builder into the `man` target.
1540
1541```nix
1542{
1543 outputs = [
1544 "out"
1545 "doc"
1546 "man"
1547 ];
1548
1549 # Use multiple builders
1550 sphinxBuilders = [
1551 "singlehtml"
1552 "man"
1553 ];
1554}
1555```
1556
1557Overwrite `sphinxRoot` when the hook is unable to find your
1558documentation source root.
1559
1560```nix
1561{
1562 # Configure sphinxRoot for uncommon paths
1563 sphinxRoot = "weird/docs/path";
1564}
1565```
1566
1567The hook is also available to packages outside the python ecosystem by
1568referencing it using `sphinxHook` from top-level.
1569
1570### Organising your packages {#organising-your-packages}
1571
1572So far we discussed how you can use Python on Nix, and how you can develop with
1573it. We've looked at how you write expressions to package Python packages, and we
1574looked at how you can create environments in which specified packages are
1575available.
1576
1577At some point you'll likely have multiple packages which you would
1578like to be able to use in different projects. In order to minimise unnecessary
1579duplication we now look at how you can maintain a repository with your
1580own packages. The important functions here are `import` and `callPackage`.
1581
1582### Including a derivation using `callPackage` {#including-a-derivation-using-callpackage}
1583
1584Earlier we created a Python environment using [`withPackages`](#python.withpackages-function), and included the
1585`toolz` package via a `let` expression.
1586Let's split the package definition from the environment definition.
1587
1588We first create a function that builds `toolz` in `~/path/to/toolz/release.nix`
1589
1590```nix
1591{
1592 lib,
1593 buildPythonPackage,
1594 fetchPypi,
1595 setuptools,
1596}:
1597
1598buildPythonPackage rec {
1599 pname = "toolz";
1600 version = "0.10.0";
1601 pyproject = true;
1602
1603 src = fetchPypi {
1604 inherit pname version;
1605 hash = "sha256-CP3V73yWSArRHBLUct4hrNMjWZlvaaUlkpm1QP66RWA=";
1606 };
1607
1608 build-system = [ setuptools ];
1609
1610 meta = {
1611 changelog = "https://github.com/pytoolz/toolz/releases/tag/${version}";
1612 homepage = "https://github.com/pytoolz/toolz/";
1613 description = "List processing tools and functional utilities";
1614 license = lib.licenses.bsd3;
1615 };
1616}
1617```
1618
1619It takes an argument [`buildPythonPackage`](#buildpythonpackage-function). We now call this function using
1620`callPackage` in the definition of our environment
1621
1622```nix
1623with import <nixpkgs> { };
1624
1625(
1626 let
1627 toolz = callPackage /path/to/toolz/release.nix {
1628 buildPythonPackage = python3Packages.buildPythonPackage;
1629 };
1630 in
1631 python3.withPackages (ps: [
1632 ps.numpy
1633 toolz
1634 ])
1635).env
1636```
1637
1638Important to remember is that the Python version for which the package is made
1639depends on the `python` derivation that is passed to [`buildPythonPackage`](#buildpythonpackage-function). Nix
1640tries to automatically pass arguments when possible, which is why generally you
1641don't explicitly define which `python` derivation should be used. In the above
1642example we use [`buildPythonPackage`](#buildpythonpackage-function) that is part of the set `python3Packages`,
1643and in this case the `python3` interpreter is automatically used.
1644
1645## FAQ {#faq}
1646
1647### How to solve circular dependencies? {#how-to-solve-circular-dependencies}
1648
1649Consider the packages `A` and `B` that depend on each other. When packaging `B`,
1650a solution is to override package `A` not to depend on `B` as an input. The same
1651should also be done when packaging `A`.
1652
1653### How to override a Python package? {#how-to-override-a-python-package}
1654
1655We can override the interpreter and pass `packageOverrides`. In the following
1656example we rename the `pandas` package and build it.
1657
1658```nix
1659with import <nixpkgs> { };
1660
1661(
1662 let
1663 python =
1664 let
1665 packageOverrides = self: super: {
1666 pandas = super.pandas.overridePythonAttrs (old: {
1667 name = "foo";
1668 });
1669 };
1670 in
1671 pkgs.python310.override { inherit packageOverrides; };
1672
1673 in
1674 python.withPackages (ps: [ ps.pandas ])
1675).env
1676```
1677
1678Using `nix-build` on this expression will build an environment that contains the
1679package `pandas` but with the new name `foo`.
1680
1681All packages in the package set will use the renamed package. A typical use case
1682is to switch to another version of a certain package. For example, in the
1683Nixpkgs repository we have multiple versions of `django` and `scipy`. In the
1684following example we use a different version of `scipy` and create an
1685environment that uses it. All packages in the Python package set will now use
1686the updated `scipy` version.
1687
1688```nix
1689with import <nixpkgs> { };
1690
1691(
1692 let
1693 packageOverrides = self: super: { scipy = super.scipy_0_17; };
1694 in
1695 (pkgs.python310.override { inherit packageOverrides; }).withPackages (ps: [ ps.blaze ])
1696).env
1697```
1698
1699The requested package `blaze` depends on `pandas` which itself depends on `scipy`.
1700
1701If you want the whole of Nixpkgs to use your modifications, then you can use
1702`overlays` as explained in this manual. In the following example we build a
1703`inkscape` using a different version of `numpy`.
1704
1705```nix
1706let
1707 pkgs = import <nixpkgs> { };
1708 newpkgs = import pkgs.path {
1709 overlays = [
1710 (self: super: {
1711 python310 =
1712 let
1713 packageOverrides = python-self: python-super: {
1714 numpy = python-super.numpy_1_18;
1715 };
1716 in
1717 super.python310.override { inherit packageOverrides; };
1718 })
1719 ];
1720 };
1721in
1722newpkgs.inkscape
1723```
1724
1725### `python setup.py bdist_wheel` cannot create .whl {#python-setup.py-bdist_wheel-cannot-create-.whl}
1726
1727Executing `python setup.py bdist_wheel` in a `nix-shell`fails with
1728
1729```
1730ValueError: ZIP does not support timestamps before 1980
1731```
1732
1733This is because files from the Nix store (which have a timestamp of the UNIX
1734epoch of January 1, 1970) are included in the .ZIP, but .ZIP archives follow the
1735DOS convention of counting timestamps from 1980.
1736
1737The command `bdist_wheel` reads the `SOURCE_DATE_EPOCH` environment variable,
1738which `nix-shell` sets to 1. Unsetting this variable or giving it a value
1739corresponding to 1980 or later enables building wheels.
1740
1741Use 1980 as timestamp:
1742
1743```shell
1744nix-shell --run "SOURCE_DATE_EPOCH=315532800 python3 setup.py bdist_wheel"
1745```
1746
1747or the current time:
1748
1749```shell
1750nix-shell --run "SOURCE_DATE_EPOCH=$(date +%s) python3 setup.py bdist_wheel"
1751```
1752
1753or unset `SOURCE_DATE_EPOCH`:
1754
1755```shell
1756nix-shell --run "unset SOURCE_DATE_EPOCH; python3 setup.py bdist_wheel"
1757```
1758
1759### `install_data` / `data_files` problems {#install_data-data_files-problems}
1760
1761If you get the following error:
1762
1763```
1764could not create '/nix/store/6l1bvljpy8gazlsw2aw9skwwp4pmvyxw-python-2.7.8/etc':
1765Permission denied
1766```
1767
1768This is a [known bug](https://github.com/pypa/setuptools/issues/130) in
1769`setuptools`. Setuptools `install_data` does not respect `--prefix`. An example
1770of such package using the feature is `pkgs/tools/X11/xpra/default.nix`.
1771
1772As workaround install it as an extra `preInstall` step:
1773
1774```shell
1775${python.pythonOnBuildForHost.interpreter} setup.py install_data --install-dir=$out --root=$out
1776sed -i '/ = data\_files/d' setup.py
1777```
1778
1779### Rationale of non-existent global site-packages {#rationale-of-non-existent-global-site-packages}
1780
1781On most operating systems a global `site-packages` is maintained. This however
1782becomes problematic if you want to run multiple Python versions or have multiple
1783versions of certain libraries for your projects. Generally, you would solve such
1784issues by creating virtual environments using `virtualenv`.
1785
1786On Nix each package has an isolated dependency tree which, in the case of
1787Python, guarantees the right versions of the interpreter and libraries or
1788packages are available. There is therefore no need to maintain a global `site-packages`.
1789
1790If you want to create a Python environment for development, then the recommended
1791method is to use `nix-shell`, either with or without the [`python.buildEnv`](#python.buildenv-function)
1792function.
1793
1794### How to consume Python modules using pip in a virtual environment like I am used to on other Operating Systems? {#how-to-consume-python-modules-using-pip-in-a-virtual-environment-like-i-am-used-to-on-other-operating-systems}
1795
1796While this approach is not very idiomatic from Nix perspective, it can still be
1797useful when dealing with pre-existing projects or in situations where it's not
1798feasible or desired to write derivations for all required dependencies.
1799
1800This is an example of a `default.nix` for a `nix-shell`, which allows to consume
1801a virtual environment created by `venv`, and install Python modules through
1802`pip` the traditional way.
1803
1804Create this `default.nix` file, together with a `requirements.txt` and
1805execute `nix-shell`.
1806
1807```nix
1808with import <nixpkgs> { };
1809
1810let
1811 pythonPackages = python3Packages;
1812in
1813pkgs.mkShell rec {
1814 name = "impurePythonEnv";
1815 venvDir = "./.venv";
1816 buildInputs = [
1817 # A Python interpreter including the 'venv' module is required to bootstrap
1818 # the environment.
1819 pythonPackages.python
1820
1821 # This executes some shell code to initialize a venv in $venvDir before
1822 # dropping into the shell
1823 pythonPackages.venvShellHook
1824
1825 # Those are dependencies that we would like to use from nixpkgs, which will
1826 # add them to PYTHONPATH and thus make them accessible from within the venv.
1827 pythonPackages.numpy
1828 pythonPackages.requests
1829
1830 # In this particular example, in order to compile any binary extensions they may
1831 # require, the Python modules listed in the hypothetical requirements.txt need
1832 # the following packages to be installed locally:
1833 taglib
1834 openssl
1835 git
1836 libxml2
1837 libxslt
1838 libzip
1839 zlib
1840 ];
1841
1842 # Run this command, only after creating the virtual environment
1843 postVenvCreation = ''
1844 unset SOURCE_DATE_EPOCH
1845 pip install -r requirements.txt
1846 '';
1847
1848 # Now we can execute any commands within the virtual environment.
1849 # This is optional and can be left out to run pip manually.
1850 postShellHook = ''
1851 # allow pip to install wheels
1852 unset SOURCE_DATE_EPOCH
1853 '';
1854
1855}
1856```
1857
1858In case the supplied venvShellHook is insufficient, or when Python 2 support is
1859needed, you can define your own shell hook and adapt to your needs like in the
1860following example:
1861
1862```nix
1863with import <nixpkgs> { };
1864
1865let
1866 venvDir = "./.venv";
1867 pythonPackages = python3Packages;
1868in
1869pkgs.mkShell rec {
1870 name = "impurePythonEnv";
1871 buildInputs = [
1872 pythonPackages.python
1873 # Needed when using python 2.7
1874 # pythonPackages.virtualenv
1875 # ...
1876 ];
1877
1878 # This is very close to how venvShellHook is implemented, but
1879 # adapted to use 'virtualenv'
1880 shellHook = ''
1881 SOURCE_DATE_EPOCH=$(date +%s)
1882
1883 if [ -d "${venvDir}" ]; then
1884 echo "Skipping venv creation, '${venvDir}' already exists"
1885 else
1886 echo "Creating new venv environment in path: '${venvDir}'"
1887 # Note that the module venv was only introduced in python 3, so for 2.7
1888 # this needs to be replaced with a call to virtualenv
1889 ${pythonPackages.python.interpreter} -m venv "${venvDir}"
1890 fi
1891
1892 # Under some circumstances it might be necessary to add your virtual
1893 # environment to PYTHONPATH, which you can do here too;
1894 # PYTHONPATH=$PWD/${venvDir}/${pythonPackages.python.sitePackages}/:$PYTHONPATH
1895
1896 source "${venvDir}/bin/activate"
1897
1898 # As in the previous example, this is optional.
1899 pip install -r requirements.txt
1900 '';
1901}
1902```
1903
1904Note that the `pip install` is an imperative action. So every time `nix-shell`
1905is executed it will attempt to download the Python modules listed in
1906requirements.txt. However these will be cached locally within the `virtualenv`
1907folder and not downloaded again.
1908
1909### How to override a Python package from `configuration.nix`? {#how-to-override-a-python-package-from-configuration.nix}
1910
1911If you need to change a package's attribute(s) from `configuration.nix` you could do:
1912
1913```nix
1914{
1915 nixpkgs.config.packageOverrides = super: {
1916 python3 = super.python3.override {
1917 packageOverrides = python-self: python-super: {
1918 twisted = python-super.twisted.overridePythonAttrs (oldAttrs: {
1919 src = super.fetchPypi {
1920 pname = "Twisted";
1921 version = "19.10.0";
1922 hash = "sha256-c5S6fycq5yKnTz2Wnc9Zm8TvCTvDkgOHSKSQ8XJKUV0=";
1923 extension = "tar.bz2";
1924 };
1925 });
1926 };
1927 };
1928 };
1929}
1930```
1931
1932`python3Packages.twisted` is now globally overridden.
1933All packages and also all NixOS services that reference `twisted`
1934(such as `services.buildbot-worker`) now use the new definition.
1935Note that `python-super` refers to the old package set and `python-self`
1936to the new, overridden version.
1937
1938To modify only a Python package set instead of a whole Python derivation, use
1939this snippet:
1940
1941```nix
1942{
1943 myPythonPackages = python3Packages.override { overrides = self: super: { twisted = <...>; }; };
1944}
1945```
1946
1947### How to override a Python package using overlays? {#how-to-override-a-python-package-using-overlays}
1948
1949Use the following overlay template:
1950
1951```nix
1952self: super: {
1953 python = super.python.override {
1954 packageOverrides = python-self: python-super: {
1955 twisted = python-super.twisted.overrideAttrs (oldAttrs: {
1956 src = super.fetchPypi {
1957 pname = "Twisted";
1958 version = "19.10.0";
1959 hash = "sha256-c5S6fycq5yKnTz2Wnc9Zm8TvCTvDkgOHSKSQ8XJKUV0=";
1960 extension = "tar.bz2";
1961 };
1962 });
1963 };
1964 };
1965}
1966```
1967
1968### How to override a Python package for all Python versions using extensions? {#how-to-override-a-python-package-for-all-python-versions-using-extensions}
1969
1970The following overlay overrides the call to [`buildPythonPackage`](#buildpythonpackage-function) for the
1971`foo` package for all interpreters by appending a Python extension to the
1972`pythonPackagesExtensions` list of extensions.
1973
1974```nix
1975final: prev: {
1976 pythonPackagesExtensions = prev.pythonPackagesExtensions ++ [
1977 (python-final: python-prev: {
1978 foo = python-prev.foo.overridePythonAttrs (oldAttrs: {
1979 # ...
1980 });
1981 })
1982 ];
1983}
1984```
1985
1986### How to use Intel’s MKL with numpy and scipy? {#how-to-use-intels-mkl-with-numpy-and-scipy}
1987
1988MKL can be configured using an overlay. See the section "[Using overlays to
1989configure alternatives](#sec-overlays-alternatives-blas-lapack)".
1990
1991### What inputs do `setup_requires`, `install_requires` and `tests_require` map to? {#what-inputs-do-setup_requires-install_requires-and-tests_require-map-to}
1992
1993In a `setup.py` or `setup.cfg` it is common to declare dependencies:
1994
1995* `setup_requires` corresponds to `build-system`
1996* `install_requires` corresponds to `dependencies`
1997* `tests_require` corresponds to [`nativeCheckInputs`](#var-stdenv-nativeCheckInputs)
1998
1999### How to enable interpreter optimizations? {#optimizations}
2000
2001The Python interpreters are by default not built with optimizations enabled, because
2002the builds are in that case not reproducible. To enable optimizations, override the
2003interpreter of interest, e.g using
2004
2005```nix
2006let
2007 pkgs = import ./. { };
2008 mypython = pkgs.python3.override {
2009 enableOptimizations = true;
2010 reproducibleBuild = false;
2011 self = mypython;
2012 };
2013in
2014mypython
2015```
2016
2017### How to add optional dependencies? {#python-optional-dependencies}
2018
2019Some packages define optional dependencies for additional features. With
2020`setuptools` this is called `extras_require` and `flit` calls it
2021`extras-require`, while PEP 621 calls these `optional-dependencies`.
2022
2023```nix
2024{
2025 optional-dependencies = {
2026 complete = [ distributed ];
2027 };
2028}
2029```
2030
2031and letting the package requiring the extra add the list to its dependencies
2032
2033```nix
2034{
2035 dependencies = [
2036 # ...
2037 ]
2038 ++ dask.optional-dependencies.complete;
2039}
2040```
2041
2042This method is using `passthru`, meaning that changing `optional-dependencies` of a package won't cause it to rebuild.
2043
2044Note this method is preferred over adding parameters to builders, as that can
2045result in packages depending on different variants and thereby causing
2046collisions.
2047
2048::: {.note}
2049The `optional-dependencies` attribute should only be used for dependency groups
2050as defined in package metadata. If a package gracefully handles missing
2051dependencies in runtime but doesn't advertise it through package metadata, then
2052these dependencies should not be listed at all. (One may still have to list
2053them in `nativeCheckInputs` to pass test suite.)
2054:::
2055
2056### How to contribute a Python package to nixpkgs? {#tools}
2057
2058Packages inside nixpkgs must use the [`buildPythonPackage`](#buildpythonpackage-function) or [`buildPythonApplication`](#buildpythonapplication-function) function directly,
2059because we can only provide security support for non-vendored dependencies.
2060
2061We recommend [nix-init](https://github.com/nix-community/nix-init) for creating new python packages within nixpkgs,
2062as it already prefetches the source, parses dependencies for common formats and prefills most things in `meta`.
2063When using the tool, pull from the original source repository instead of PyPI, if possible.
2064
2065See also [contributing section](#contributing).
2066
2067### Are Python interpreters built deterministically? {#deterministic-builds}
2068
2069The Python interpreters are now built deterministically. Minor modifications had
2070to be made to the interpreters in order to generate deterministic bytecode. This
2071has security implications and is relevant for those using Python in a
2072`nix-shell`.
2073
2074When the environment variable `DETERMINISTIC_BUILD` is set, all bytecode will
2075have timestamp 1. The [`buildPythonPackage`](#buildpythonpackage-function) function sets `DETERMINISTIC_BUILD=1`
2076and [PYTHONHASHSEED=0](https://docs.python.org/3.13/using/cmdline.html#envvar-PYTHONHASHSEED).
2077Both are also exported in `nix-shell`.
2078
2079### How to provide automatic tests to Python packages? {#automatic-tests}
2080
2081It is recommended to test packages as part of the build process.
2082Source distributions (`sdist`) often include test files, but not always.
2083
2084The best practice today is to pass a test hook (e.g. pytestCheckHook, unittestCheckHook) into nativeCheckInputs.
2085This will reconfigure the checkPhase to make use of that particular test framework.
2086Occasionally packages don't make use of a common test framework, which may then require a custom checkPhase.
2087
2088#### Common issues {#common-issues}
2089
2090* Tests that attempt to access `$HOME` can be fixed by using `writableTmpDirAsHomeHook` in
2091 `nativeCheckInputs`, which sets up a writable temporary directory as the home directory. Alternatively,
2092 you can achieve the same effect manually (e.g. in `preCheck`) with: `export HOME=$(mktemp -d)`.
2093* Compiling with Cython causes tests to fail with a `ModuleNotLoadedError`.
2094 This can be fixed with two changes in the derivation: 1) replacing `pytest` with
2095 `pytestCheckHook` and 2) adding a `preCheck` containing `cd $out` to run
2096 tests within the built output.
2097
2098## Contributing {#contributing}
2099
2100### Contributing guidelines {#contributing-guidelines}
2101
2102The following rules are desired to be respected:
2103
2104* Python libraries are called from `python-packages.nix` and packaged with
2105 [`buildPythonPackage`](#buildpythonpackage-function). The expression of a library should be in
2106 `pkgs/development/python-modules/<name>/default.nix`.
2107* Python applications live outside of `python-packages.nix` and are packaged
2108 with [`buildPythonApplication`](#buildpythonapplication-function).
2109* Make sure libraries build for all Python interpreters.
2110 If it fails to build on some Python versions, consider disabling them by setting `disable = pythonAtLeast "3.x"` along with a comment.
2111* The two parameters, `pyproject` and `build-system` are set to avoid the legacy setuptools/distutils build.
2112* Only unversioned attributes (e.g. `pydantic`, but not `pypdantic_1`) can be included in `dependencies`,
2113 since due to `PYTHONPATH` limitations we can only ever support a single version for libraries
2114 without running into duplicate module name conflicts.
2115* The version restrictions of `dependencies` can be relaxed by [`pythonRelaxDepsHook`](#using-pythonrelaxdepshook).
2116* Make sure the tests are enabled using for example [`pytestCheckHook`](#using-pytestcheckhook) and, in the case of
2117 libraries, are passing for all interpreters. If certain tests fail they can be
2118 disabled individually. Try to avoid disabling the tests altogether. In any
2119 case, when you disable tests, leave a comment explaining not only _what_ the failure
2120 is but _why_ the test failure can be ignored for safe distribution with nixpkgs.
2121* `pythonImportsCheck` is set. This is still a good smoke test even if `pytestCheckHook` is set.
2122* `meta.platforms` takes the default value in many cases.
2123 It does not need to be set explicitly unless the package requires a specific platform.
2124* The file is formatted with `nixfmt-rfc-style`.
2125* Commit names of Python libraries must reflect that they are Python
2126 libraries (e.g. `python3Packages.numpy: 1.11 -> 1.12` rather than `numpy: 1.11 -> 1.12`).
2127 See also [`pkgs/README.md`](https://github.com/NixOS/nixpkgs/blob/master/pkgs/README.md#commit-conventions).
2128* Attribute names in `python-packages.nix` as well as `pname`s should match the
2129 library's name on PyPI, but be normalized according to [PEP
2130 0503](https://www.python.org/dev/peps/pep-0503/#normalized-names). This means
2131 that characters should be converted to lowercase and `.` and `_` should be
2132 replaced by a single `-` (foo-bar-baz instead of Foo__Bar.baz).
2133 If necessary, `pname` has to be given a different value within `fetchPypi`.
2134* It's generally preferable to fetch `src` directly from the repo and not from
2135 PyPI. Use `fetchPypi` when there's a clear technical reason to do so.
2136* Packages from sources such as GitHub and GitLab that do not exist on PyPI
2137 should not use a name that is already used on PyPI. When possible, they should
2138 use the package repository name prefixed with the owner (e.g. organization) name
2139 and using a `-` as delimiter.
2140* Attribute names in `python-packages.nix` should be sorted alphanumerically to
2141 avoid merge conflicts and ease locating attributes.
2142* Non-python runtime dependencies should be added via explicit wrapping or
2143 patching (using e.g. `substituteInPlace`), rather than through propagation via
2144 `dependencies`/`propagatedBuildInputs`, to reduce clutter in `$PATH`.
2145
2146This list is useful for reviewers as well as for self-checking when submitting packages.
2147
2148## Package set maintenance {#python-package-set-maintenance}
2149
2150The whole Python package set has a lot of packages that do not see regular
2151updates, because they either are a very fragile component in the Python
2152ecosystem, like for example the `hypothesis` package, or packages that have
2153no maintainer, so maintenance falls back to the package set maintainers.
2154
2155### Updating packages in bulk {#python-package-bulk-updates}
2156
2157A tool to bulk-update numerous Python libraries is available in the
2158repository at `maintainers/scripts/update-python-libraries`.
2159
2160It can quickly update minor or major versions for all packages selected
2161and create update commits, and supports the `fetchPypi`, `fetchurl` and
2162`fetchFromGitHub` fetchers. When updating lots of packages that are
2163hosted on GitHub, exporting a `GITHUB_API_TOKEN` is highly recommended.
2164
2165Updating packages in bulk leads to lots of breakages, which is why a
2166stabilization period on the `python-updates` branch is required.
2167
2168If a package is fragile and often breaks during these bulks updates, it
2169may be reasonable to set `passthru.skipBulkUpdate = true` in the
2170derivation. This decision should not be made on a whim and should
2171always be supported by a qualifying comment.
2172
2173Once the branch is sufficiently stable it should normally be merged
2174into the `staging` branch.
2175
2176An exemplary call to update all python libraries between minor versions
2177would be:
2178
2179```ShellSession
2180$ maintainers/scripts/update-python-libraries --target minor --commit --use-pkgs-prefix pkgs/development/python-modules/**/default.nix
2181```
2182
2183## CPython Update Schedule {#python-cpython-update-schedule}
2184
2185With [PEP 602](https://www.python.org/dev/peps/pep-0602/), CPython now
2186follows a yearly release cadence. In nixpkgs, all supported interpreters
2187are made available, but only the most recent two
2188interpreters package sets are built; this is a compromise between being
2189the latest interpreter, and what the majority of the Python packages support.
2190
2191New CPython interpreters are released in October. Generally, it takes some
2192time for the majority of active Python projects to support the latest stable
2193interpreter. To help ease the migration for Nixpkgs users
2194between Python interpreters the schedule below will be used:
2195
2196| When | Event |
2197| --- | --- |
2198| After YY.11 Release | Bump CPython package set window. The latest and previous latest stable should now be built. |
2199| After YY.05 Release | Bump default CPython interpreter to latest stable. |
2200
2201In practice, this means that the Python community will have had a stable interpreter
2202for ~2 months before attempting to update the package set. And this will
2203allow for ~7 months for Python applications to support the latest interpreter.