pyrox.dev
1# Python {#python}
2
3## User Guide {#user-guide}
4
5### Using Python {#using-python}
6
7#### Overview {#overview}
8
9Several versions of the Python interpreter are available on Nix, as well as a
10high amount of packages. The attribute `python3` refers to the default
11interpreter, which is currently CPython 3.10. The attribute `python` refers to
12CPython 2.7 for backwards-compatibility. It is also possible to refer to
13specific versions, e.g. `python311` refers to CPython 3.11, and `pypy` refers to
14the default PyPy interpreter.
15
16Python is used a lot, and in different ways. This affects also how it is
17packaged. In the case of Python on Nix, an important distinction is made between
18whether the package is considered primarily an application, or whether it should
19be used as a library, i.e., of primary interest are the modules in
20`site-packages` that should be importable.
21
22In the Nixpkgs tree Python applications can be found throughout, depending on
23what they do, and are called from the main package set. Python libraries,
24however, are in separate sets, with one set per interpreter version.
25
26The interpreters have several common attributes. One of these attributes is
27`pkgs`, which is a package set of Python libraries for this specific
28interpreter. E.g., the `toolz` package corresponding to the default interpreter
29is `python.pkgs.toolz`, and the CPython 3.11 version is `python311.pkgs.toolz`.
30The main package set contains aliases to these package sets, e.g.
31`pythonPackages` refers to `python.pkgs` and `python311Packages` to
32`python311.pkgs`.
33
34#### Installing Python and packages {#installing-python-and-packages}
35
36The Nix and NixOS manuals explain how packages are generally installed. In the
37case of Python and Nix, it is important to make a distinction between whether the
38package is considered an application or a library.
39
40Applications on Nix are typically installed into your user profile imperatively
41using `nix-env -i`, and on NixOS declaratively by adding the package name to
42`environment.systemPackages` in `/etc/nixos/configuration.nix`. Dependencies
43such as libraries are automatically installed and should not be installed
44explicitly.
45
46The same goes for Python applications. Python applications can be installed in
47your profile, and will be wrapped to find their exact library dependencies,
48without impacting other applications or polluting your user environment.
49
50But Python libraries you would like to use for development cannot be installed,
51at least not individually, because they won't be able to find each other
52resulting in import errors. Instead, it is possible to create an environment
53with `python.buildEnv` or `python.withPackages` where the interpreter and other
54executables are wrapped to be able to find each other and all of the modules.
55
56In the following examples we will start by creating a simple, ad-hoc environment
57with a nix-shell that has `numpy` and `toolz` in Python 3.11; then we will create
58a re-usable environment in a single-file Python script; then we will create a
59full Python environment for development with this same environment.
60
61Philosophically, this should be familiar to users who are used to a `venv` style
62of development: individual projects create their own Python environments without
63impacting the global environment or each other.
64
65#### Ad-hoc temporary Python environment with `nix-shell` {#ad-hoc-temporary-python-environment-with-nix-shell}
66
67The simplest way to start playing with the way nix wraps and sets up Python
68environments is with `nix-shell` at the cmdline. These environments create a
69temporary shell session with a Python and a *precise* list of packages (plus
70their runtime dependencies), with no other Python packages in the Python
71interpreter's scope.
72
73To create a Python 3.11 session with `numpy` and `toolz` available, run:
74
75```sh
76$ nix-shell -p 'python311.withPackages(ps: with ps; [ numpy toolz ])'
77```
78
79By default `nix-shell` will start a `bash` session with this interpreter in our
80`PATH`, so if we then run:
81
82```Python console
83[nix-shell:~/src/nixpkgs]$ python3
84Python 3.11.3 (main, Apr 4 2023, 22:36:41) [GCC 12.2.0] on linux
85Type "help", "copyright", "credits" or "license" for more information.
86>>> import numpy; import toolz
87```
88
89Note that no other modules are in scope, even if they were imperatively
90installed into our user environment as a dependency of a Python application:
91
92```Python console
93>>> import requests
94Traceback (most recent call last):
95 File "<stdin>", line 1, in <module>
96ModuleNotFoundError: No module named 'requests'
97```
98
99We can add as many additional modules onto the `nix-shell` as we need, and we
100will still get 1 wrapped Python interpreter. We can start the interpreter
101directly like so:
102
103```sh
104$ nix-shell -p "python311.withPackages (ps: with ps; [ numpy toolz requests ])" --run python3
105this derivation will be built:
106 /nix/store/r19yf5qgfiakqlhkgjahbg3zg79549n4-python3-3.11.2-env.drv
107building '/nix/store/r19yf5qgfiakqlhkgjahbg3zg79549n4-python3-3.11.2-env.drv'...
108created 273 symlinks in user environment
109Python 3.11.2 (main, Feb 7 2023, 13:52:42) [GCC 12.2.0] on linux
110Type "help", "copyright", "credits" or "license" for more information.
111>>> import requests
112>>>
113```
114
115Notice that this time it built a new Python environment, which now includes
116`requests`. Building an environment just creates wrapper scripts that expose the
117selected dependencies to the interpreter while re-using the actual modules. This
118means if any other env has installed `requests` or `numpy` in a different
119context, we don't need to recompile them -- we just recompile the wrapper script
120that sets up an interpreter pointing to them. This matters much more for "big"
121modules like `pytorch` or `tensorflow`.
122
123Module names usually match their names on [pypi.org](https://pypi.org/), but
124you can use the [Nixpkgs search website](https://nixos.org/nixos/packages.html)
125to find them as well (along with non-python packages).
126
127At this point we can create throwaway experimental Python environments with
128arbitrary dependencies. This is a good way to get a feel for how the Python
129interpreter and dependencies work in Nix and NixOS, but to do some actual
130development, we'll want to make it a bit more persistent.
131
132##### Running Python scripts and using `nix-shell` as shebang {#running-python-scripts-and-using-nix-shell-as-shebang}
133
134Sometimes, we have a script whose header looks like this:
135
136```python
137#!/usr/bin/env python3
138import numpy as np
139a = np.array([1,2])
140b = np.array([3,4])
141print(f"The dot product of {a} and {b} is: {np.dot(a, b)}")
142```
143
144Executing this script requires a `python3` that has `numpy`. Using what we learned
145in the previous section, we could startup a shell and just run it like so:
146
147```ShellSession
148$ nix-shell -p 'python311.withPackages (ps: with ps; [ numpy ])' --run 'python3 foo.py'
149The dot product of [1 2] and [3 4] is: 11
150```
151
152But if we maintain the script ourselves, and if there are more dependencies, it
153may be nice to encode those dependencies in source to make the script re-usable
154without that bit of knowledge. That can be done by using `nix-shell` as a
155[shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)), like so:
156
157```python
158#!/usr/bin/env nix-shell
159#!nix-shell -i python3 -p "python3.withPackages(ps: [ ps.numpy ])"
160import numpy as np
161a = np.array([1,2])
162b = np.array([3,4])
163print(f"The dot product of {a} and {b} is: {np.dot(a, b)}")
164```
165
166Then we simply execute it, without requiring any environment setup at all!
167
168```sh
169$ ./foo.py
170The dot product of [1 2] and [3 4] is: 11
171```
172
173If the dependencies are not available on the host where `foo.py` is executed, it
174will build or download them from a Nix binary cache prior to starting up, prior
175that it is executed on a machine with a multi-user nix installation.
176
177This provides a way to ship a self bootstrapping Python script, akin to a
178statically linked binary, where it can be run on any machine (provided nix is
179installed) without having to assume that `numpy` is installed globally on the
180system.
181
182By default it is pulling the import checkout of Nixpkgs itself from our nix
183channel, which is nice as it cache aligns with our other package builds, but we
184can make it fully reproducible by pinning the `nixpkgs` import:
185
186```python
187#!/usr/bin/env nix-shell
188#!nix-shell -i python3 -p "python3.withPackages (ps: [ ps.numpy ])"
189#!nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/e51209796c4262bfb8908e3d6d72302fe4e96f5f.tar.gz
190import numpy as np
191a = np.array([1,2])
192b = np.array([3,4])
193print(f"The dot product of {a} and {b} is: {np.dot(a, b)}")
194```
195
196This will execute with the exact same versions of Python 3.10, numpy, and system
197dependencies a year from now as it does today, because it will always use
198exactly git commit `e51209796c4262bfb8908e3d6d72302fe4e96f5f` of Nixpkgs for all
199of the package versions.
200
201This is also a great way to ensure the script executes identically on different
202servers.
203
204##### Load environment from `.nix` expression {#load-environment-from-.nix-expression}
205
206We've now seen how to create an ad-hoc temporary shell session, and how to
207create a single script with Python dependencies, but in the course of normal
208development we're usually working in an entire package repository.
209
210As explained in the Nix manual, `nix-shell` can also load an expression from a
211`.nix` file. Say we want to have Python 3.11, `numpy` and `toolz`, like before,
212in an environment. We can add a `shell.nix` file describing our dependencies:
213
214```nix
215with import <nixpkgs> {};
216(python311.withPackages (ps: with ps; [
217 numpy
218 toolz
219])).env
220```
221
222And then at the command line, just typing `nix-shell` produces the same
223environment as before. In a normal project, we'll likely have many more
224dependencies; this can provide a way for developers to share the environments
225with each other and with CI builders.
226
227What's happening here?
228
2291. We begin with importing the Nix Packages collections. `import <nixpkgs>`
230 imports the `<nixpkgs>` function, `{}` calls it and the `with` statement
231 brings all attributes of `nixpkgs` in the local scope. These attributes form
232 the main package set.
2332. Then we create a Python 3.11 environment with the `withPackages` function, as before.
2343. The `withPackages` function expects us to provide a function as an argument
235 that takes the set of all Python packages and returns a list of packages to
236 include in the environment. Here, we select the packages `numpy` and `toolz`
237 from the package set.
238
239To combine this with `mkShell` you can:
240
241```nix
242with import <nixpkgs> {};
243let
244 pythonEnv = python311.withPackages (ps: [
245 ps.numpy
246 ps.toolz
247 ]);
248in mkShell {
249 packages = [
250 pythonEnv
251
252 black
253 mypy
254
255 libffi
256 openssl
257 ];
258}
259```
260
261This will create a unified environment that has not just our Python interpreter
262and its Python dependencies, but also tools like `black` or `mypy` and libraries
263like `libffi` the `openssl` in scope. This is generic and can span any number of
264tools or languages across the Nixpkgs ecosystem.
265
266##### Installing environments globally on the system {#installing-environments-globally-on-the-system}
267
268Up to now, we've been creating environments scoped to an ad-hoc shell session,
269or a single script, or a single project. This is generally advisable, as it
270avoids pollution across contexts.
271
272However, sometimes we know we will often want a Python with some basic packages,
273and want this available without having to enter into a shell or build context.
274This can be useful to have things like vim/emacs editors and plugins or shell
275tools "just work" without having to set them up, or when running other software
276that expects packages to be installed globally.
277
278To create your own custom environment, create a file in `~/.config/nixpkgs/overlays/`
279that looks like this:
280
281```nix
282# ~/.config/nixpkgs/overlays/myEnv.nix
283self: super: {
284 myEnv = super.buildEnv {
285 name = "myEnv";
286 paths = [
287 # A Python 3 interpreter with some packages
288 (self.python3.withPackages (
289 ps: with ps; [
290 pyflakes
291 pytest
292 black
293 ]
294 ))
295
296 # Some other packages we'd like as part of this env
297 self.mypy
298 self.black
299 self.ripgrep
300 self.tmux
301 ];
302 };
303}
304```
305
306You can then build and install this to your profile with:
307
308```sh
309nix-env -iA myEnv
310```
311
312One limitation of this is that you can only have 1 Python env installed
313globally, since they conflict on the `python` to load out of your `PATH`.
314
315If you get a conflict or prefer to keep the setup clean, you can have `nix-env`
316atomically *uninstall* all other imperatively installed packages and replace
317your profile with just `myEnv` by using the `--replace` flag.
318
319##### Environment defined in `/etc/nixos/configuration.nix` {#environment-defined-in-etcnixosconfiguration.nix}
320
321For the sake of completeness, here's how to install the environment system-wide
322on NixOS.
323
324```nix
325{ # ...
326
327 environment.systemPackages = with pkgs; [
328 (python310.withPackages(ps: with ps; [ numpy toolz ]))
329 ];
330}
331```
332
333### Developing with Python {#developing-with-python}
334
335Above, we were mostly just focused on use cases and what to do to get started
336creating working Python environments in nix.
337
338Now that you know the basics to be up and running, it is time to take a step
339back and take a deeper look at how Python packages are packaged on Nix. Then,
340we will look at how you can use development mode with your code.
341
342#### Python library packages in Nixpkgs {#python-library-packages-in-nixpkgs}
343
344With Nix all packages are built by functions. The main function in Nix for
345building Python libraries is `buildPythonPackage`. Let's see how we can build the
346`toolz` package.
347
348```nix
349{ lib
350, buildPythonPackage
351, fetchPypi
352}:
353
354buildPythonPackage rec {
355 pname = "toolz";
356 version = "0.10.0";
357 format = "setuptools";
358
359 src = fetchPypi {
360 inherit pname version;
361 hash = "sha256-CP3V73yWSArRHBLUct4hrNMjWZlvaaUlkpm1QP66RWA=";
362 };
363
364 # has no tests
365 doCheck = false;
366
367 pythonImportsCheck = [
368 "toolz.itertoolz"
369 "toolz.functoolz"
370 "toolz.dicttoolz"
371 ];
372
373 meta = with lib; {
374 changelog = "https://github.com/pytoolz/toolz/releases/tag/${version}";
375 homepage = "https://github.com/pytoolz/toolz";
376 description = "List processing tools and functional utilities";
377 license = licenses.bsd3;
378 maintainers = with maintainers; [ fridh ];
379 };
380}
381```
382
383What happens here? The function `buildPythonPackage` is called and as argument
384it accepts a set. In this case the set is a recursive set, `rec`. One of the
385arguments is the name of the package, which consists of a basename (generally
386following the name on PyPi) and a version. Another argument, `src` specifies the
387source, which in this case is fetched from PyPI using the helper function
388`fetchPypi`. The argument `doCheck` is used to set whether tests should be run
389when building the package. Since there are no tests, we rely on `pythonImportsCheck`
390to test whether the package can be imported. Furthermore, we specify some meta
391information. The output of the function is a derivation.
392
393An expression for `toolz` can be found in the Nixpkgs repository. As explained
394in the introduction of this Python section, a derivation of `toolz` is available
395for each interpreter version, e.g. `python311.pkgs.toolz` refers to the `toolz`
396derivation corresponding to the CPython 3.11 interpreter.
397
398The above example works when you're directly working on
399`pkgs/top-level/python-packages.nix` in the Nixpkgs repository. Often though,
400you will want to test a Nix expression outside of the Nixpkgs tree.
401
402The following expression creates a derivation for the `toolz` package,
403and adds it along with a `numpy` package to a Python environment.
404
405```nix
406with import <nixpkgs> {};
407
408( let
409 my_toolz = python311.pkgs.buildPythonPackage rec {
410 pname = "toolz";
411 version = "0.10.0";
412 format = "setuptools";
413
414 src = fetchPypi {
415 inherit pname version;
416 hash = "sha256-CP3V73yWSArRHBLUct4hrNMjWZlvaaUlkpm1QP66RWA=";
417 };
418
419 # has no tests
420 doCheck = false;
421
422 meta = {
423 homepage = "https://github.com/pytoolz/toolz/";
424 description = "List processing tools and functional utilities";
425 # [...]
426 };
427 };
428
429 in python311.withPackages (ps: with ps; [
430 numpy
431 my_toolz
432 ])
433).env
434```
435
436Executing `nix-shell` will result in an environment in which you can use
437Python 3.11 and the `toolz` package. As you can see we had to explicitly mention
438for which Python version we want to build a package.
439
440So, what did we do here? Well, we took the Nix expression that we used earlier
441to build a Python environment, and said that we wanted to include our own
442version of `toolz`, named `my_toolz`. To introduce our own package in the scope
443of `withPackages` we used a `let` expression. You can see that we used
444`ps.numpy` to select numpy from the nixpkgs package set (`ps`). We did not take
445`toolz` from the Nixpkgs package set this time, but instead took our own version
446that we introduced with the `let` expression.
447
448#### Handling dependencies {#handling-dependencies}
449
450Our example, `toolz`, does not have any dependencies on other Python packages or
451system libraries. According to the manual, `buildPythonPackage` uses the
452arguments `buildInputs` and `propagatedBuildInputs` to specify dependencies. If
453something is exclusively a build-time dependency, then the dependency should be
454included in `buildInputs`, but if it is (also) a runtime dependency, then it
455should be added to `propagatedBuildInputs`. Test dependencies are considered
456build-time dependencies and passed to `nativeCheckInputs`.
457
458The following example shows which arguments are given to `buildPythonPackage` in
459order to build [`datashape`](https://github.com/blaze/datashape).
460
461```nix
462{ lib
463, buildPythonPackage
464, fetchPypi
465
466# dependencies
467, numpy, multipledispatch, python-dateutil
468
469# tests
470, pytest
471}:
472
473buildPythonPackage rec {
474 pname = "datashape";
475 version = "0.4.7";
476 format = "setuptools";
477
478 src = fetchPypi {
479 inherit pname version;
480 hash = "sha256-FLLvdm1MllKrgTGC6Gb0k0deZeVYvtCCLji/B7uhong=";
481 };
482
483 propagatedBuildInputs = [
484 multipledispatch
485 numpy
486 python-dateutil
487 ];
488
489 nativeCheckInputs = [
490 pytest
491 ];
492
493 meta = with lib; {
494 changelog = "https://github.com/blaze/datashape/releases/tag/${version}";
495 homepage = "https://github.com/ContinuumIO/datashape";
496 description = "A data description language";
497 license = licenses.bsd2;
498 maintainers = with maintainers; [ fridh ];
499 };
500}
501```
502
503We can see several runtime dependencies, `numpy`, `multipledispatch`, and
504`python-dateutil`. Furthermore, we have `nativeCheckInputs` with `pytest`.
505`pytest` is a test runner and is only used during the `checkPhase` and is
506therefore not added to `propagatedBuildInputs`.
507
508In the previous case we had only dependencies on other Python packages to consider.
509Occasionally you have also system libraries to consider. E.g., `lxml` provides
510Python bindings to `libxml2` and `libxslt`. These libraries are only required
511when building the bindings and are therefore added as `buildInputs`.
512
513```nix
514{ lib
515, pkgs
516, buildPythonPackage
517, fetchPypi
518}:
519
520buildPythonPackage rec {
521 pname = "lxml";
522 version = "3.4.4";
523 format = "setuptools";
524
525 src = fetchPypi {
526 inherit pname version;
527 hash = "sha256-s9NiusRxFydHzaNRMjjxFcvWxfi45jGb9ql6eJJyQJk=";
528 };
529
530 buildInputs = [
531 pkgs.libxml2
532 pkgs.libxslt
533 ];
534
535 meta = with lib; {
536 changelog = "https://github.com/lxml/lxml/releases/tag/lxml-${version}";
537 description = "Pythonic binding for the libxml2 and libxslt libraries";
538 homepage = "https://lxml.de";
539 license = licenses.bsd3;
540 maintainers = with maintainers; [ sjourdois ];
541 };
542}
543```
544
545In this example `lxml` and Nix are able to work out exactly where the relevant
546files of the dependencies are. This is not always the case.
547
548The example below shows bindings to The Fastest Fourier Transform in the West,
549commonly known as FFTW. On Nix we have separate packages of FFTW for the
550different types of floats (`"single"`, `"double"`, `"long-double"`). The
551bindings need all three types, and therefore we add all three as `buildInputs`.
552The bindings don't expect to find each of them in a different folder, and
553therefore we have to set `LDFLAGS` and `CFLAGS`.
554
555```nix
556{ lib
557, pkgs
558, buildPythonPackage
559, fetchPypi
560
561# dependencies
562, numpy
563, scipy
564}:
565
566buildPythonPackage rec {
567 pname = "pyFFTW";
568 version = "0.9.2";
569 format = "setuptools";
570
571 src = fetchPypi {
572 inherit pname version;
573 hash = "sha256-9ru2r6kwhUCaskiFoaPNuJCfCVoUL01J40byvRt4kHQ=";
574 };
575
576 buildInputs = [
577 pkgs.fftw
578 pkgs.fftwFloat
579 pkgs.fftwLongDouble
580 ];
581
582 propagatedBuildInputs = [
583 numpy
584 scipy
585 ];
586
587 preConfigure = ''
588 export LDFLAGS="-L${pkgs.fftw.dev}/lib -L${pkgs.fftwFloat.out}/lib -L${pkgs.fftwLongDouble.out}/lib"
589 export CFLAGS="-I${pkgs.fftw.dev}/include -I${pkgs.fftwFloat.dev}/include -I${pkgs.fftwLongDouble.dev}/include"
590 '';
591
592 # Tests cannot import pyfftw. pyfftw works fine though.
593 doCheck = false;
594
595 meta = with lib; {
596 changelog = "https://github.com/pyFFTW/pyFFTW/releases/tag/v${version}";
597 description = "A pythonic wrapper around FFTW, the FFT library, presenting a unified interface for all the supported transforms";
598 homepage = "http://hgomersall.github.com/pyFFTW";
599 license = with licenses; [ bsd2 bsd3 ];
600 maintainers = with maintainers; [ fridh ];
601 };
602}
603```
604
605Note also the line `doCheck = false;`, we explicitly disabled running the test-suite.
606
607#### Testing Python Packages {#testing-python-packages}
608
609It is highly encouraged to have testing as part of the package build. This
610helps to avoid situations where the package was able to build and install,
611but is not usable at runtime. Currently, all packages will use the `test`
612command provided by the setup.py (i.e. `python setup.py test`). However,
613this is currently deprecated https://github.com/pypa/setuptools/pull/1878
614and your package should provide its own checkPhase.
615
616*NOTE:* The `checkPhase` for python maps to the `installCheckPhase` on a
617normal derivation. This is due to many python packages not behaving well
618to the pre-installed version of the package. Version info, and natively
619compiled extensions generally only exist in the install directory, and
620thus can cause issues when a test suite asserts on that behavior.
621
622*NOTE:* Tests should only be disabled if they don't agree with nix
623(e.g. external dependencies, network access, flakey tests), however,
624as many tests should be enabled as possible. Failing tests can still be
625a good indication that the package is not in a valid state.
626
627#### Using pytest {#using-pytest}
628
629Pytest is the most common test runner for python repositories. A trivial
630test run would be:
631
632```
633 nativeCheckInputs = [ pytest ];
634 checkPhase = ''
635 runHook preCheck
636
637 pytest
638
639 runHook postCheck
640 '';
641```
642
643However, many repositories' test suites do not translate well to nix's build
644sandbox, and will generally need many tests to be disabled.
645
646To filter tests using pytest, one can do the following:
647
648```
649 nativeCheckInputs = [ pytest ];
650 # avoid tests which need additional data or touch network
651 checkPhase = ''
652 runHook preCheck
653
654 pytest tests/ --ignore=tests/integration -k 'not download and not update' --ignore=tests/test_failing.py
655
656 runHook postCheck
657 '';
658```
659
660`--ignore` will tell pytest to ignore that file or directory from being
661collected as part of a test run. This is useful is a file uses a package
662which is not available in nixpkgs, thus skipping that test file is much
663easier than having to create a new package.
664
665`-k` is used to define a predicate for test names. In this example, we are
666filtering out tests which contain `download` or `update` in their test case name.
667Only one `-k` argument is allowed, and thus a long predicate should be concatenated
668with “\\” and wrapped to the next line.
669
670*NOTE:* In pytest==6.0.1, the use of “\\” to continue a line (e.g. `-k 'not download \'`) has
671been removed, in this case, it's recommended to use `pytestCheckHook`.
672
673#### Using pytestCheckHook {#using-pytestcheckhook}
674
675`pytestCheckHook` is a convenient hook which will substitute the setuptools
676`test` command for a `checkPhase` which runs `pytest`. This is also beneficial
677when a package may need many items disabled to run the test suite.
678
679Using the example above, the analogous `pytestCheckHook` usage would be:
680
681```
682 nativeCheckInputs = [
683 pytestCheckHook
684 ];
685
686 # requires additional data
687 pytestFlagsArray = [
688 "tests/"
689 "--ignore=tests/integration"
690 ];
691
692 disabledTests = [
693 # touches network
694 "download"
695 "update"
696 ];
697
698 disabledTestPaths = [
699 "tests/test_failing.py"
700 ];
701```
702
703This is especially useful when tests need to be conditionally disabled,
704for example:
705
706```
707 disabledTests = [
708 # touches network
709 "download"
710 "update"
711 ] ++ lib.optionals (pythonAtLeast "3.8") [
712 # broken due to python3.8 async changes
713 "async"
714 ] ++ lib.optionals stdenv.isDarwin [
715 # can fail when building with other packages
716 "socket"
717 ];
718```
719
720Trying to concatenate the related strings to disable tests in a regular
721`checkPhase` would be much harder to read. This also enables us to comment on
722why specific tests are disabled.
723
724#### Using pythonImportsCheck {#using-pythonimportscheck}
725
726Although unit tests are highly preferred to validate correctness of a package, not
727all packages have test suites that can be run easily, and some have none at all.
728To help ensure the package still works, `pythonImportsCheck` can attempt to import
729the listed modules.
730
731```
732 pythonImportsCheck = [
733 "requests"
734 "urllib"
735 ];
736```
737
738roughly translates to:
739
740```
741 postCheck = ''
742 PYTHONPATH=$out/${python.sitePackages}:$PYTHONPATH
743 python -c "import requests; import urllib"
744 '';
745```
746
747However, this is done in its own phase, and not dependent on whether `doCheck = true;`.
748
749This can also be useful in verifying that the package doesn't assume commonly
750present packages (e.g. `setuptools`).
751
752#### Using pythonRelaxDepsHook {#using-pythonrelaxdepshook}
753
754It is common for upstream to specify a range of versions for its package
755dependencies. This makes sense, since it ensures that the package will be built
756with a subset of packages that is well tested. However, this commonly causes
757issues when packaging in Nixpkgs, because the dependencies that this package
758may need are too new or old for the package to build correctly. We also cannot
759package multiple versions of the same package since this may cause conflicts
760in `PYTHONPATH`.
761
762One way to side step this issue is to relax the dependencies. This can be done
763by either removing the package version range or by removing the package
764declaration entirely. This can be done using the `pythonRelaxDepsHook` hook. For
765example, given the following `requirements.txt` file:
766
767```
768pkg1<1.0
769pkg2
770pkg3>=1.0,<=2.0
771```
772
773we can do:
774
775```
776 nativeBuildInputs = [
777 pythonRelaxDepsHook
778 ];
779 pythonRelaxDeps = [
780 "pkg1"
781 "pkg3"
782 ];
783 pythonRemoveDeps = [
784 "pkg2"
785 ];
786```
787
788which would result in the following `requirements.txt` file:
789
790```
791pkg1
792pkg3
793```
794
795Another option is to pass `true`, that will relax/remove all dependencies, for
796example:
797
798```
799 nativeBuildInputs = [ pythonRelaxDepsHook ];
800 pythonRelaxDeps = true;
801```
802
803which would result in the following `requirements.txt` file:
804
805```
806pkg1
807pkg2
808pkg3
809```
810
811In general you should always use `pythonRelaxDeps`, because `pythonRemoveDeps`
812will convert build errors into runtime errors. However `pythonRemoveDeps` may
813still be useful in exceptional cases, and also to remove dependencies wrongly
814declared by upstream (for example, declaring `black` as a runtime dependency
815instead of a dev dependency).
816
817Keep in mind that while the examples above are done with `requirements.txt`,
818`pythonRelaxDepsHook` works by modifying the resulting wheel file, so it should
819work in any of the formats supported by `buildPythonPackage` currently,
820with the exception of `other` (see `format` in
821[`buildPythonPackage` parameters](#buildpythonpackage-parameters) for more details).
822
823#### Using unittestCheckHook {#using-unittestcheckhook}
824
825`unittestCheckHook` is a hook which will substitute the setuptools `test` command for a `checkPhase` which runs `python -m unittest discover`:
826
827```
828 nativeCheckInputs = [
829 unittestCheckHook
830 ];
831
832 unittestFlagsArray = [
833 "-s" "tests" "-v"
834 ];
835```
836
837#### Using sphinxHook {#using-sphinxhook}
838
839The `sphinxHook` is a helpful tool to build documentation and manpages
840using the popular Sphinx documentation generator.
841It is setup to automatically find common documentation source paths and
842render them using the default `html` style.
843
844```
845 outputs = [
846 "out"
847 "doc"
848 ];
849
850 nativeBuildInputs = [
851 sphinxHook
852 ];
853```
854
855The hook will automatically build and install the artifact into the
856`doc` output, if it exists. It also provides an automatic diversion
857for the artifacts of the `man` builder into the `man` target.
858
859```
860 outputs = [
861 "out"
862 "doc"
863 "man"
864 ];
865
866 # Use multiple builders
867 sphinxBuilders = [
868 "singlehtml"
869 "man"
870 ];
871```
872
873Overwrite `sphinxRoot` when the hook is unable to find your
874documentation source root.
875
876```
877 # Configure sphinxRoot for uncommon paths
878 sphinxRoot = "weird/docs/path";
879```
880
881The hook is also available to packages outside the python ecosystem by
882referencing it using `sphinxHook` from top-level.
883
884### Develop local package {#develop-local-package}
885
886As a Python developer you're likely aware of [development mode](http://setuptools.readthedocs.io/en/latest/setuptools.html#development-mode)
887(`python setup.py develop`); instead of installing the package this command
888creates a special link to the project code. That way, you can run updated code
889without having to reinstall after each and every change you make. Development
890mode is also available. Let's see how you can use it.
891
892In the previous Nix expression the source was fetched from a url. We can also
893refer to a local source instead using `src = ./path/to/source/tree;`
894
895If we create a `shell.nix` file which calls `buildPythonPackage`, and if `src`
896is a local source, and if the local source has a `setup.py`, then development
897mode is activated.
898
899In the following example, we create a simple environment that has a Python 3.11
900version of our package in it, as well as its dependencies and other packages we
901like to have in the environment, all specified with `propagatedBuildInputs`.
902Indeed, we can just add any package we like to have in our environment to
903`propagatedBuildInputs`.
904
905```nix
906with import <nixpkgs> {};
907with python311Packages;
908
909buildPythonPackage rec {
910 name = "mypackage";
911 src = ./path/to/package/source;
912 propagatedBuildInputs = [
913 pytest
914 numpy
915 pkgs.libsndfile
916 ];
917}
918```
919
920It is important to note that due to how development mode is implemented on Nix
921it is not possible to have multiple packages simultaneously in development mode.
922
923### Organising your packages {#organising-your-packages}
924
925So far we discussed how you can use Python on Nix, and how you can develop with
926it. We've looked at how you write expressions to package Python packages, and we
927looked at how you can create environments in which specified packages are
928available.
929
930At some point you'll likely have multiple packages which you would
931like to be able to use in different projects. In order to minimise unnecessary
932duplication we now look at how you can maintain a repository with your
933own packages. The important functions here are `import` and `callPackage`.
934
935### Including a derivation using `callPackage` {#including-a-derivation-using-callpackage}
936
937Earlier we created a Python environment using `withPackages`, and included the
938`toolz` package via a `let` expression.
939Let's split the package definition from the environment definition.
940
941We first create a function that builds `toolz` in `~/path/to/toolz/release.nix`
942
943```nix
944{ lib
945, buildPythonPackage
946}:
947
948buildPythonPackage rec {
949 pname = "toolz";
950 version = "0.10.0";
951 format = "setuptools";
952
953 src = fetchPypi {
954 inherit pname version;
955 hash = "sha256-CP3V73yWSArRHBLUct4hrNMjWZlvaaUlkpm1QP66RWA=";
956 };
957
958 meta = with lib; {
959 changelog = "https://github.com/pytoolz/toolz/releases/tag/${version}";
960 homepage = "https://github.com/pytoolz/toolz/";
961 description = "List processing tools and functional utilities";
962 license = licenses.bsd3;
963 maintainers = with maintainers; [ fridh ];
964 };
965}
966```
967
968It takes an argument `buildPythonPackage`. We now call this function using
969`callPackage` in the definition of our environment
970
971```nix
972with import <nixpkgs> {};
973
974( let
975 toolz = callPackage /path/to/toolz/release.nix {
976 buildPythonPackage = python310
977Packages.buildPythonPackage;
978 };
979 in python310.withPackages (ps: [
980 ps.numpy
981 toolz
982 ])
983).env
984```
985
986Important to remember is that the Python version for which the package is made
987depends on the `python` derivation that is passed to `buildPythonPackage`. Nix
988tries to automatically pass arguments when possible, which is why generally you
989don't explicitly define which `python` derivation should be used. In the above
990example we use `buildPythonPackage` that is part of the set `python3Packages`,
991and in this case the `python3` interpreter is automatically used.
992
993## Reference {#reference}
994
995### Interpreters {#interpreters}
996
997Versions 2.7, 3.8, 3.9, 3.10 and 3.11 of the CPython interpreter are available
998as respectively `python27`, python38`, `python39`, `python310` and `python311`.
999The aliases `python2` and `python3` correspond to respectively `python27` and
1000`python310`. The attribute `python` maps to `python2`. The PyPy interpreters
1001compatible with Python 2.7 and 3 are available as `pypy27` and `pypy3`, with
1002aliases `pypy2` mapping to `pypy27` and `pypy` mapping to `pypy2`. The Nix
1003expressions for the interpreters can be found in
1004`pkgs/development/interpreters/python`.
1005
1006All packages depending on any Python interpreter get appended
1007`out/{python.sitePackages}` to `$PYTHONPATH` if such directory
1008exists.
1009
1010#### Missing `tkinter` module standard library {#missing-tkinter-module-standard-library}
1011
1012To reduce closure size the `Tkinter`/`tkinter` is available as a separate package, `pythonPackages.tkinter`.
1013
1014#### Attributes on interpreters packages {#attributes-on-interpreters-packages}
1015
1016Each interpreter has the following attributes:
1017
1018- `libPrefix`. Name of the folder in `${python}/lib/` for corresponding interpreter.
1019- `interpreter`. Alias for `${python}/bin/${executable}`.
1020- `buildEnv`. Function to build python interpreter environments with extra packages bundled together. See section *python.buildEnv function* for usage and documentation.
1021- `withPackages`. Simpler interface to `buildEnv`. See section *python.withPackages function* for usage and documentation.
1022- `sitePackages`. Alias for `lib/${libPrefix}/site-packages`.
1023- `executable`. Name of the interpreter executable, e.g. `python3.10`.
1024- `pkgs`. Set of Python packages for that specific interpreter. The package set can be modified by overriding the interpreter and passing `packageOverrides`.
1025
1026### Optimizations {#optimizations}
1027
1028The Python interpreters are by default not built with optimizations enabled, because
1029the builds are in that case not reproducible. To enable optimizations, override the
1030interpreter of interest, e.g using
1031
1032```
1033let
1034 pkgs = import ./. {};
1035 mypython = pkgs.python3.override {
1036 enableOptimizations = true;
1037 reproducibleBuild = false;
1038 self = mypython;
1039 };
1040in mypython
1041```
1042
1043### Building packages and applications {#building-packages-and-applications}
1044
1045Python libraries and applications that use `setuptools` or
1046`distutils` are typically built with respectively the `buildPythonPackage` and
1047`buildPythonApplication` functions. These two functions also support installing a `wheel`.
1048
1049All Python packages reside in `pkgs/top-level/python-packages.nix` and all
1050applications elsewhere. In case a package is used as both a library and an
1051application, then the package should be in `pkgs/top-level/python-packages.nix`
1052since only those packages are made available for all interpreter versions. The
1053preferred location for library expressions is in
1054`pkgs/development/python-modules`. It is important that these packages are
1055called from `pkgs/top-level/python-packages.nix` and not elsewhere, to guarantee
1056the right version of the package is built.
1057
1058Based on the packages defined in `pkgs/top-level/python-packages.nix` an
1059attribute set is created for each available Python interpreter. The available
1060sets are
1061
1062* `pkgs.python27Packages`
1063* `pkgs.python3Packages`
1064* `pkgs.python38Packages`
1065* `pkgs.python39Packages`
1066* `pkgs.python310Packages`
1067* `pkgs.python311Packages`
1068* `pkgs.pypyPackages`
1069
1070and the aliases
1071
1072* `pkgs.python2Packages` pointing to `pkgs.python27Packages`
1073* `pkgs.python3Packages` pointing to `pkgs.python310Packages`
1074* `pkgs.pythonPackages` pointing to `pkgs.python2Packages`
1075
1076#### `buildPythonPackage` function {#buildpythonpackage-function}
1077
1078The `buildPythonPackage` function is implemented in
1079`pkgs/development/interpreters/python/mk-python-derivation.nix`
1080using setup hooks.
1081
1082The following is an example:
1083
1084```nix
1085{ lib
1086, buildPythonPackage
1087, fetchPypi
1088
1089# build-system
1090, setuptools-scm
1091
1092# dependencies
1093, attrs
1094, pluggy
1095, py
1096, setuptools
1097, six
1098
1099# tests
1100, hypothesis
1101 }:
1102
1103buildPythonPackage rec {
1104 pname = "pytest";
1105 version = "3.3.1";
1106 format = "setuptools";
1107
1108 src = fetchPypi {
1109 inherit pname version;
1110 hash = "sha256-z4Q23FnYaVNG/NOrKW3kZCXsqwDWQJbOvnn7Ueyy65M=";
1111 };
1112
1113 postPatch = ''
1114 # don't test bash builtins
1115 rm testing/test_argcomplete.py
1116 '';
1117
1118 nativeBuildInputs = [
1119 setuptools-scm
1120 ];
1121
1122 propagatedBuildInputs = [
1123 attrs
1124 py
1125 setuptools
1126 six
1127 pluggy
1128 ];
1129
1130 nativeCheckInputs = [
1131 hypothesis
1132 ];
1133
1134 meta = with lib; {
1135 changelog = "https://github.com/pytest-dev/pytest/releases/tag/${version}";
1136 description = "Framework for writing tests";
1137 homepage = "https://github.com/pytest-dev/pytest";
1138 license = licenses.mit;
1139 maintainers = with maintainers; [ domenkozar lovek323 madjar lsix ];
1140 };
1141}
1142```
1143
1144The `buildPythonPackage` mainly does four things:
1145
1146* In the `buildPhase`, it calls `${python.pythonForBuild.interpreter} setup.py bdist_wheel` to
1147 build a wheel binary zipfile.
1148* In the `installPhase`, it installs the wheel file using `pip install *.whl`.
1149* In the `postFixup` phase, the `wrapPythonPrograms` bash function is called to
1150 wrap all programs in the `$out/bin/*` directory to include `$PATH`
1151 environment variable and add dependent libraries to script's `sys.path`.
1152* In the `installCheck` phase, `${python.interpreter} setup.py test` is run.
1153
1154By default tests are run because `doCheck = true`. Test dependencies, like
1155e.g. the test runner, should be added to `nativeCheckInputs`.
1156
1157By default `meta.platforms` is set to the same value
1158as the interpreter unless overridden otherwise.
1159
1160##### `buildPythonPackage` parameters {#buildpythonpackage-parameters}
1161
1162All parameters from `stdenv.mkDerivation` function are still supported. The
1163following are specific to `buildPythonPackage`:
1164
1165* `catchConflicts ? true`: If `true`, abort package build if a package name
1166 appears more than once in dependency tree. Default is `true`.
1167* `disabled ? false`: If `true`, package is not built for the particular Python
1168 interpreter version.
1169* `dontWrapPythonPrograms ? false`: Skip wrapping of Python programs.
1170* `permitUserSite ? false`: Skip setting the `PYTHONNOUSERSITE` environment
1171 variable in wrapped programs.
1172* `format ? "setuptools"`: Format of the source. Valid options are
1173 `"setuptools"`, `"pyproject"`, `"flit"`, `"wheel"`, and `"other"`.
1174 `"setuptools"` is for when the source has a `setup.py` and `setuptools` is
1175 used to build a wheel, `flit`, in case `flit` should be used to build a wheel,
1176 and `wheel` in case a wheel is provided. Use `other` when a custom
1177 `buildPhase` and/or `installPhase` is needed.
1178* `makeWrapperArgs ? []`: A list of strings. Arguments to be passed to
1179 `makeWrapper`, which wraps generated binaries. By default, the arguments to
1180 `makeWrapper` set `PATH` and `PYTHONPATH` environment variables before calling
1181 the binary. Additional arguments here can allow a developer to set environment
1182 variables which will be available when the binary is run. For example,
1183 `makeWrapperArgs = ["--set FOO BAR" "--set BAZ QUX"]`.
1184* `namePrefix`: Prepends text to `${name}` parameter. In case of libraries, this
1185 defaults to `"python3.8-"` for Python 3.8, etc., and in case of applications
1186 to `""`.
1187* `pipInstallFlags ? []`: A list of strings. Arguments to be passed to `pip
1188 install`. To pass options to `python setup.py install`, use
1189 `--install-option`. E.g., `pipInstallFlags=["--install-option='--cpp_implementation'"]`.
1190* `pythonPath ? []`: List of packages to be added into `$PYTHONPATH`. Packages
1191 in `pythonPath` are not propagated (contrary to `propagatedBuildInputs`).
1192* `preShellHook`: Hook to execute commands before `shellHook`.
1193* `postShellHook`: Hook to execute commands after `shellHook`.
1194* `removeBinByteCode ? true`: Remove bytecode from `/bin`. Bytecode is only
1195 created when the filenames end with `.py`.
1196* `setupPyGlobalFlags ? []`: List of flags passed to `setup.py` command.
1197* `setupPyBuildFlags ? []`: List of flags passed to `setup.py build_ext` command.
1198
1199The `stdenv.mkDerivation` function accepts various parameters for describing
1200build inputs (see "Specifying dependencies"). The following are of special
1201interest for Python packages, either because these are primarily used, or
1202because their behaviour is different:
1203
1204* `nativeBuildInputs ? []`: Build-time only dependencies. Typically executables
1205 as well as the items listed in `setup_requires`.
1206* `buildInputs ? []`: Build and/or run-time dependencies that need to be
1207 compiled for the host machine. Typically non-Python libraries which are being
1208 linked.
1209* `nativeCheckInputs ? []`: Dependencies needed for running the `checkPhase`. These
1210 are added to `nativeBuildInputs` when `doCheck = true`. Items listed in
1211 `tests_require` go here.
1212* `propagatedBuildInputs ? []`: Aside from propagating dependencies,
1213 `buildPythonPackage` also injects code into and wraps executables with the
1214 paths included in this list. Items listed in `install_requires` go here.
1215
1216##### Overriding Python packages {#overriding-python-packages}
1217
1218The `buildPythonPackage` function has a `overridePythonAttrs` method that can be
1219used to override the package. In the following example we create an environment
1220where we have the `blaze` package using an older version of `pandas`. We
1221override first the Python interpreter and pass `packageOverrides` which contains
1222the overrides for packages in the package set.
1223
1224```nix
1225with import <nixpkgs> {};
1226
1227(let
1228 python = let
1229 packageOverrides = self: super: {
1230 pandas = super.pandas.overridePythonAttrs(old: rec {
1231 version = "0.19.1";
1232 src = fetchPypi {
1233 pname = "pandas";
1234 inherit version;
1235 hash = "sha256-JQn+rtpy/OA2deLszSKEuxyttqBzcAil50H+JDHUdCE=";
1236 };
1237 });
1238 };
1239 in pkgs.python3.override {inherit packageOverrides; self = python;};
1240
1241in python.withPackages(ps: [ ps.blaze ])).env
1242```
1243
1244#### Optional extra dependencies {#python-optional-dependencies}
1245
1246Some packages define optional dependencies for additional features. With
1247`setuptools` this is called `extras_require` and `flit` calls it
1248`extras-require`, while PEP 621 calls these `optional-dependencies`. A
1249method for supporting this is by declaring the extras of a package in its
1250`passthru`, e.g. in case of the package `dask`
1251
1252```nix
1253passthru.optional-dependencies = {
1254 complete = [ distributed ];
1255};
1256```
1257
1258and letting the package requiring the extra add the list to its dependencies
1259
1260```nix
1261propagatedBuildInputs = [
1262 ...
1263] ++ dask.optional-dependencies.complete;
1264```
1265
1266Note this method is preferred over adding parameters to builders, as that can
1267result in packages depending on different variants and thereby causing
1268collisions.
1269
1270#### `buildPythonApplication` function {#buildpythonapplication-function}
1271
1272The `buildPythonApplication` function is practically the same as
1273`buildPythonPackage`. The main purpose of this function is to build a Python
1274package where one is interested only in the executables, and not importable
1275modules. For that reason, when adding this package to a `python.buildEnv`, the
1276modules won't be made available.
1277
1278Another difference is that `buildPythonPackage` by default prefixes the names of
1279the packages with the version of the interpreter. Because this is irrelevant for
1280applications, the prefix is omitted.
1281
1282When packaging a Python application with `buildPythonApplication`, it should be
1283called with `callPackage` and passed `python` or `pythonPackages` (possibly
1284specifying an interpreter version), like this:
1285
1286```nix
1287{ lib
1288, python3
1289, fetchPypi
1290}:
1291
1292python3.pkgs.buildPythonApplication rec {
1293 pname = "luigi";
1294 version = "2.7.9";
1295 format = "setuptools";
1296
1297 src = fetchPypi {
1298 inherit pname version;
1299 hash = "sha256-Pe229rT0aHwA98s+nTHQMEFKZPo/yw6sot8MivFDvAw=";
1300 };
1301
1302 propagatedBuildInputs = with python3.pkgs; [
1303 tornado
1304 python-daemon
1305 ];
1306
1307 meta = with lib; {
1308 ...
1309 };
1310}
1311```
1312
1313This is then added to `all-packages.nix` just as any other application would be.
1314
1315```nix
1316luigi = callPackage ../applications/networking/cluster/luigi { };
1317```
1318
1319Since the package is an application, a consumer doesn't need to care about
1320Python versions or modules, which is why they don't go in `pythonPackages`.
1321
1322#### `toPythonApplication` function {#topythonapplication-function}
1323
1324A distinction is made between applications and libraries, however, sometimes a
1325package is used as both. In this case the package is added as a library to
1326`python-packages.nix` and as an application to `all-packages.nix`. To reduce
1327duplication the `toPythonApplication` can be used to convert a library to an
1328application.
1329
1330The Nix expression shall use `buildPythonPackage` and be called from
1331`python-packages.nix`. A reference shall be created from `all-packages.nix` to
1332the attribute in `python-packages.nix`, and the `toPythonApplication` shall be
1333applied to the reference:
1334
1335```nix
1336youtube-dl = with pythonPackages; toPythonApplication youtube-dl;
1337```
1338
1339#### `toPythonModule` function {#topythonmodule-function}
1340
1341In some cases, such as bindings, a package is created using
1342`stdenv.mkDerivation` and added as attribute in `all-packages.nix`. The Python
1343bindings should be made available from `python-packages.nix`. The
1344`toPythonModule` function takes a derivation and makes certain Python-specific
1345modifications.
1346
1347```nix
1348opencv = toPythonModule (pkgs.opencv.override {
1349 enablePython = true;
1350 pythonPackages = self;
1351});
1352```
1353
1354Do pay attention to passing in the right Python version!
1355
1356#### `python.buildEnv` function {#python.buildenv-function}
1357
1358Python environments can be created using the low-level `pkgs.buildEnv` function.
1359This example shows how to create an environment that has the Pyramid Web Framework.
1360Saving the following as `default.nix`
1361
1362```nix
1363with import <nixpkgs> {};
1364
1365python.buildEnv.override {
1366 extraLibs = [ pythonPackages.pyramid ];
1367 ignoreCollisions = true;
1368}
1369```
1370
1371and running `nix-build` will create
1372
1373```
1374/nix/store/cf1xhjwzmdki7fasgr4kz6di72ykicl5-python-2.7.8-env
1375```
1376
1377with wrapped binaries in `bin/`.
1378
1379You can also use the `env` attribute to create local environments with needed
1380packages installed. This is somewhat comparable to `virtualenv`. For example,
1381running `nix-shell` with the following `shell.nix`
1382
1383```nix
1384with import <nixpkgs> {};
1385
1386(python3.buildEnv.override {
1387 extraLibs = with python3Packages; [
1388 numpy
1389 requests
1390 ];
1391}).env
1392```
1393
1394will drop you into a shell where Python will have the
1395specified packages in its path.
1396
1397##### `python.buildEnv` arguments {#python.buildenv-arguments}
1398
1399
1400* `extraLibs`: List of packages installed inside the environment.
1401* `postBuild`: Shell command executed after the build of environment.
1402* `ignoreCollisions`: Ignore file collisions inside the environment (default is `false`).
1403* `permitUserSite`: Skip setting the `PYTHONNOUSERSITE` environment variable in
1404 wrapped binaries in the environment.
1405
1406#### `python.withPackages` function {#python.withpackages-function}
1407
1408The `python.withPackages` function provides a simpler interface to the `python.buildEnv` functionality.
1409It takes a function as an argument that is passed the set of python packages and returns the list
1410of the packages to be included in the environment. Using the `withPackages` function, the previous
1411example for the Pyramid Web Framework environment can be written like this:
1412
1413```nix
1414with import <nixpkgs> {};
1415
1416python.withPackages (ps: [ ps.pyramid ])
1417```
1418
1419`withPackages` passes the correct package set for the specific interpreter
1420version as an argument to the function. In the above example, `ps` equals
1421`pythonPackages`. But you can also easily switch to using python3:
1422
1423```nix
1424with import <nixpkgs> {};
1425
1426python3.withPackages (ps: [ ps.pyramid ])
1427```
1428
1429Now, `ps` is set to `python3Packages`, matching the version of the interpreter.
1430
1431As `python.withPackages` simply uses `python.buildEnv` under the hood, it also
1432supports the `env` attribute. The `shell.nix` file from the previous section can
1433thus be also written like this:
1434
1435```nix
1436with import <nixpkgs> {};
1437
1438(python3.withPackages (ps: with ps; [
1439 numpy
1440 requests
1441])).env
1442```
1443
1444In contrast to `python.buildEnv`, `python.withPackages` does not support the
1445more advanced options such as `ignoreCollisions = true` or `postBuild`. If you
1446need them, you have to use `python.buildEnv`.
1447
1448Python 2 namespace packages may provide `__init__.py` that collide. In that case
1449`python.buildEnv` should be used with `ignoreCollisions = true`.
1450
1451#### Setup hooks {#setup-hooks}
1452
1453The following are setup hooks specifically for Python packages. Most of these
1454are used in `buildPythonPackage`.
1455
1456- `eggUnpackhook` to move an egg to the correct folder so it can be installed
1457 with the `eggInstallHook`
1458- `eggBuildHook` to skip building for eggs.
1459- `eggInstallHook` to install eggs.
1460- `flitBuildHook` to build a wheel using `flit`.
1461- `pipBuildHook` to build a wheel using `pip` and PEP 517. Note a build system
1462 (e.g. `setuptools` or `flit`) should still be added as `nativeBuildInput`.
1463- `pipInstallHook` to install wheels.
1464- `pytestCheckHook` to run tests with `pytest`. See [example usage](#using-pytestcheckhook).
1465- `pythonCatchConflictsHook` to check whether a Python package is not already existing.
1466- `pythonImportsCheckHook` to check whether importing the listed modules works.
1467- `pythonRelaxDepsHook` will relax Python dependencies restrictions for the package.
1468 See [example usage](#using-pythonrelaxdepshook).
1469- `pythonRemoveBinBytecode` to remove bytecode from the `/bin` folder.
1470- `setuptoolsBuildHook` to build a wheel using `setuptools`.
1471- `setuptoolsCheckHook` to run tests with `python setup.py test`.
1472- `sphinxHook` to build documentation and manpages using Sphinx.
1473- `venvShellHook` to source a Python 3 `venv` at the `venvDir` location. A
1474 `venv` is created if it does not yet exist. `postVenvCreation` can be used to
1475 to run commands only after venv is first created.
1476- `wheelUnpackHook` to move a wheel to the correct folder so it can be installed
1477 with the `pipInstallHook`.
1478- `unittestCheckHook` will run tests with `python -m unittest discover`. See [example usage](#using-unittestcheckhook).
1479
1480### Development mode {#development-mode}
1481
1482Development or editable mode is supported. To develop Python packages
1483`buildPythonPackage` has additional logic inside `shellPhase` to run `pip
1484install -e . --prefix $TMPDIR/`for the package.
1485
1486Warning: `shellPhase` is executed only if `setup.py` exists.
1487
1488Given a `default.nix`:
1489
1490```nix
1491with import <nixpkgs> {};
1492
1493pythonPackages.buildPythonPackage {
1494 name = "myproject";
1495 buildInputs = with pythonPackages; [ pyramid ];
1496
1497 src = ./.;
1498}
1499```
1500
1501Running `nix-shell` with no arguments should give you the environment in which
1502the package would be built with `nix-build`.
1503
1504Shortcut to setup environments with C headers/libraries and Python packages:
1505
1506```shell
1507nix-shell -p pythonPackages.pyramid zlib libjpeg git
1508```
1509
1510Note: There is a boolean value `lib.inNixShell` set to `true` if nix-shell is invoked.
1511
1512### Tools {#tools}
1513
1514Packages inside nixpkgs are written by hand. However many tools exist in
1515community to help save time. No tool is preferred at the moment.
1516
1517- [pypi2nix](https://github.com/nix-community/pypi2nix): Generate Nix
1518 expressions for your Python project. Note that [sharing derivations from
1519 pypi2nix with nixpkgs is possible but not
1520 encouraged](https://github.com/nix-community/pypi2nix/issues/222#issuecomment-443497376).
1521- [nixpkgs-pytools](https://github.com/nix-community/nixpkgs-pytools)
1522- [poetry2nix](https://github.com/nix-community/poetry2nix)
1523
1524### Deterministic builds {#deterministic-builds}
1525
1526The Python interpreters are now built deterministically. Minor modifications had
1527to be made to the interpreters in order to generate deterministic bytecode. This
1528has security implications and is relevant for those using Python in a
1529`nix-shell`.
1530
1531When the environment variable `DETERMINISTIC_BUILD` is set, all bytecode will
1532have timestamp 1. The `buildPythonPackage` function sets `DETERMINISTIC_BUILD=1`
1533and [PYTHONHASHSEED=0](https://docs.python.org/3.11/using/cmdline.html#envvar-PYTHONHASHSEED).
1534Both are also exported in `nix-shell`.
1535
1536### Automatic tests {#automatic-tests}
1537
1538It is recommended to test packages as part of the build process.
1539Source distributions (`sdist`) often include test files, but not always.
1540
1541By default the command `python setup.py test` is run as part of the
1542`checkPhase`, but often it is necessary to pass a custom `checkPhase`. An
1543example of such a situation is when `py.test` is used.
1544
1545#### Common issues {#common-issues}
1546
1547* Non-working tests can often be deselected. By default `buildPythonPackage`
1548 runs `python setup.py test`. which is deprecated. Most Python modules however
1549 do follow the standard test protocol where the pytest runner can be used
1550 instead. `pytest` supports the `-k` and `--ignore` parameters to ignore test
1551 methods or classes as well as whole files. For `pytestCheckHook` these are
1552 conveniently exposed as `disabledTests` and `disabledTestPaths` respectively.
1553
1554 ```nix
1555 buildPythonPackage {
1556 # ...
1557 nativeCheckInputs = [
1558 pytestCheckHook
1559 ];
1560
1561 disabledTests = [
1562 "function_name"
1563 "other_function"
1564 ];
1565
1566 disabledTestPaths = [
1567 "this/file.py"
1568 ];
1569 }
1570 ```
1571
1572* Tests that attempt to access `$HOME` can be fixed by using the following
1573 work-around before running tests (e.g. `preCheck`): `export HOME=$(mktemp -d)`
1574
1575## FAQ {#faq}
1576
1577### How to solve circular dependencies? {#how-to-solve-circular-dependencies}
1578
1579Consider the packages `A` and `B` that depend on each other. When packaging `B`,
1580a solution is to override package `A` not to depend on `B` as an input. The same
1581should also be done when packaging `A`.
1582
1583### How to override a Python package? {#how-to-override-a-python-package}
1584
1585We can override the interpreter and pass `packageOverrides`. In the following
1586example we rename the `pandas` package and build it.
1587
1588```nix
1589with import <nixpkgs> {};
1590
1591(let
1592 python = let
1593 packageOverrides = self: super: {
1594 pandas = super.pandas.overridePythonAttrs(old: {name="foo";});
1595 };
1596 in pkgs.python310.override {
1597 inherit packageOverrides;
1598 };
1599
1600in python.withPackages (ps: [
1601 ps.pandas
1602])).env
1603```
1604
1605Using `nix-build` on this expression will build an environment that contains the
1606package `pandas` but with the new name `foo`.
1607
1608All packages in the package set will use the renamed package. A typical use case
1609is to switch to another version of a certain package. For example, in the
1610Nixpkgs repository we have multiple versions of `django` and `scipy`. In the
1611following example we use a different version of `scipy` and create an
1612environment that uses it. All packages in the Python package set will now use
1613the updated `scipy` version.
1614
1615```nix
1616with import <nixpkgs> {};
1617
1618( let
1619 packageOverrides = self: super: {
1620 scipy = super.scipy_0_17;
1621 };
1622 in (pkgs.python310.override {
1623 inherit packageOverrides;
1624 }).withPackages (ps: [
1625 ps.blaze
1626 ])
1627).env
1628```
1629
1630The requested package `blaze` depends on `pandas` which itself depends on `scipy`.
1631
1632If you want the whole of Nixpkgs to use your modifications, then you can use
1633`overlays` as explained in this manual. In the following example we build a
1634`inkscape` using a different version of `numpy`.
1635
1636```nix
1637let
1638 pkgs = import <nixpkgs> {};
1639 newpkgs = import pkgs.path { overlays = [ (self: super: {
1640 python310 = let
1641 packageOverrides = python-self: python-super: {
1642 numpy = python-super.numpy_1_18;
1643 };
1644 in super.python310.override {inherit packageOverrides;};
1645 } ) ]; };
1646in newpkgs.inkscape
1647```
1648
1649### `python setup.py bdist_wheel` cannot create .whl {#python-setup.py-bdist_wheel-cannot-create-.whl}
1650
1651Executing `python setup.py bdist_wheel` in a `nix-shell`fails with
1652
1653```
1654ValueError: ZIP does not support timestamps before 1980
1655```
1656
1657This is because files from the Nix store (which have a timestamp of the UNIX
1658epoch of January 1, 1970) are included in the .ZIP, but .ZIP archives follow the
1659DOS convention of counting timestamps from 1980.
1660
1661The command `bdist_wheel` reads the `SOURCE_DATE_EPOCH` environment variable,
1662which `nix-shell` sets to 1. Unsetting this variable or giving it a value
1663corresponding to 1980 or later enables building wheels.
1664
1665Use 1980 as timestamp:
1666
1667```shell
1668nix-shell --run "SOURCE_DATE_EPOCH=315532800 python3 setup.py bdist_wheel"
1669```
1670
1671or the current time:
1672
1673```shell
1674nix-shell --run "SOURCE_DATE_EPOCH=$(date +%s) python3 setup.py bdist_wheel"
1675```
1676
1677or unset `SOURCE_DATE_EPOCH`:
1678
1679```shell
1680nix-shell --run "unset SOURCE_DATE_EPOCH; python3 setup.py bdist_wheel"
1681```
1682
1683### `install_data` / `data_files` problems {#install_data-data_files-problems}
1684
1685If you get the following error:
1686
1687```
1688could not create '/nix/store/6l1bvljpy8gazlsw2aw9skwwp4pmvyxw-python-2.7.8/etc':
1689Permission denied
1690```
1691
1692This is a [known bug](https://github.com/pypa/setuptools/issues/130) in
1693`setuptools`. Setuptools `install_data` does not respect `--prefix`. An example
1694of such package using the feature is `pkgs/tools/X11/xpra/default.nix`.
1695
1696As workaround install it as an extra `preInstall` step:
1697
1698```shell
1699${python.pythonForBuild.interpreter} setup.py install_data --install-dir=$out --root=$out
1700sed -i '/ = data\_files/d' setup.py
1701```
1702
1703### Rationale of non-existent global site-packages {#rationale-of-non-existent-global-site-packages}
1704
1705On most operating systems a global `site-packages` is maintained. This however
1706becomes problematic if you want to run multiple Python versions or have multiple
1707versions of certain libraries for your projects. Generally, you would solve such
1708issues by creating virtual environments using `virtualenv`.
1709
1710On Nix each package has an isolated dependency tree which, in the case of
1711Python, guarantees the right versions of the interpreter and libraries or
1712packages are available. There is therefore no need to maintain a global `site-packages`.
1713
1714If you want to create a Python environment for development, then the recommended
1715method is to use `nix-shell`, either with or without the `python.buildEnv`
1716function.
1717
1718### How to consume Python modules using pip in a virtual environment like I am used to on other Operating Systems? {#how-to-consume-python-modules-using-pip-in-a-virtual-environment-like-i-am-used-to-on-other-operating-systems}
1719
1720While this approach is not very idiomatic from Nix perspective, it can still be
1721useful when dealing with pre-existing projects or in situations where it's not
1722feasible or desired to write derivations for all required dependencies.
1723
1724This is an example of a `default.nix` for a `nix-shell`, which allows to consume
1725a virtual environment created by `venv`, and install Python modules through
1726`pip` the traditional way.
1727
1728Create this `default.nix` file, together with a `requirements.txt` and simply
1729execute `nix-shell`.
1730
1731```nix
1732with import <nixpkgs> { };
1733
1734let
1735 pythonPackages = python3Packages;
1736in pkgs.mkShell rec {
1737 name = "impurePythonEnv";
1738 venvDir = "./.venv";
1739 buildInputs = [
1740 # A Python interpreter including the 'venv' module is required to bootstrap
1741 # the environment.
1742 pythonPackages.python
1743
1744 # This executes some shell code to initialize a venv in $venvDir before
1745 # dropping into the shell
1746 pythonPackages.venvShellHook
1747
1748 # Those are dependencies that we would like to use from nixpkgs, which will
1749 # add them to PYTHONPATH and thus make them accessible from within the venv.
1750 pythonPackages.numpy
1751 pythonPackages.requests
1752
1753 # In this particular example, in order to compile any binary extensions they may
1754 # require, the Python modules listed in the hypothetical requirements.txt need
1755 # the following packages to be installed locally:
1756 taglib
1757 openssl
1758 git
1759 libxml2
1760 libxslt
1761 libzip
1762 zlib
1763 ];
1764
1765 # Run this command, only after creating the virtual environment
1766 postVenvCreation = ''
1767 unset SOURCE_DATE_EPOCH
1768 pip install -r requirements.txt
1769 '';
1770
1771 # Now we can execute any commands within the virtual environment.
1772 # This is optional and can be left out to run pip manually.
1773 postShellHook = ''
1774 # allow pip to install wheels
1775 unset SOURCE_DATE_EPOCH
1776 '';
1777
1778}
1779```
1780
1781In case the supplied venvShellHook is insufficient, or when Python 2 support is
1782needed, you can define your own shell hook and adapt to your needs like in the
1783following example:
1784
1785```nix
1786with import <nixpkgs> { };
1787
1788let
1789 venvDir = "./.venv";
1790 pythonPackages = python3Packages;
1791in pkgs.mkShell rec {
1792 name = "impurePythonEnv";
1793 buildInputs = [
1794 pythonPackages.python
1795 # Needed when using python 2.7
1796 # pythonPackages.virtualenv
1797 # ...
1798 ];
1799
1800 # This is very close to how venvShellHook is implemented, but
1801 # adapted to use 'virtualenv'
1802 shellHook = ''
1803 SOURCE_DATE_EPOCH=$(date +%s)
1804
1805 if [ -d "${venvDir}" ]; then
1806 echo "Skipping venv creation, '${venvDir}' already exists"
1807 else
1808 echo "Creating new venv environment in path: '${venvDir}'"
1809 # Note that the module venv was only introduced in python 3, so for 2.7
1810 # this needs to be replaced with a call to virtualenv
1811 ${pythonPackages.python.interpreter} -m venv "${venvDir}"
1812 fi
1813
1814 # Under some circumstances it might be necessary to add your virtual
1815 # environment to PYTHONPATH, which you can do here too;
1816 # PYTHONPATH=$PWD/${venvDir}/${pythonPackages.python.sitePackages}/:$PYTHONPATH
1817
1818 source "${venvDir}/bin/activate"
1819
1820 # As in the previous example, this is optional.
1821 pip install -r requirements.txt
1822 '';
1823}
1824```
1825
1826Note that the `pip install` is an imperative action. So every time `nix-shell`
1827is executed it will attempt to download the Python modules listed in
1828requirements.txt. However these will be cached locally within the `virtualenv`
1829folder and not downloaded again.
1830
1831### How to override a Python package from `configuration.nix`? {#how-to-override-a-python-package-from-configuration.nix}
1832
1833If you need to change a package's attribute(s) from `configuration.nix` you could do:
1834
1835```nix
1836 nixpkgs.config.packageOverrides = super: {
1837 python3 = super.python3.override {
1838 packageOverrides = python-self: python-super: {
1839 twisted = python-super.twisted.overridePythonAttrs (oldAttrs: {
1840 src = super.fetchPypi {
1841 pname = "Twisted";
1842 version = "19.10.0";
1843 hash = "sha256-c5S6fycq5yKnTz2Wnc9Zm8TvCTvDkgOHSKSQ8XJKUV0=";
1844 extension = "tar.bz2";
1845 };
1846 });
1847 };
1848 };
1849 };
1850```
1851
1852`pythonPackages.twisted` is now globally overridden.
1853All packages and also all NixOS services that reference `twisted`
1854(such as `services.buildbot-worker`) now use the new definition.
1855Note that `python-super` refers to the old package set and `python-self`
1856to the new, overridden version.
1857
1858To modify only a Python package set instead of a whole Python derivation, use
1859this snippet:
1860
1861```nix
1862 myPythonPackages = pythonPackages.override {
1863 overrides = self: super: {
1864 twisted = ...;
1865 };
1866 }
1867```
1868
1869### How to override a Python package using overlays? {#how-to-override-a-python-package-using-overlays}
1870
1871Use the following overlay template:
1872
1873```nix
1874self: super: {
1875 python = super.python.override {
1876 packageOverrides = python-self: python-super: {
1877 twisted = python-super.twisted.overrideAttrs (oldAttrs: {
1878 src = super.fetchPypi {
1879 pname = "Twisted";
1880 version = "19.10.0";
1881 hash = "sha256-c5S6fycq5yKnTz2Wnc9Zm8TvCTvDkgOHSKSQ8XJKUV0=";
1882 extension = "tar.bz2";
1883 };
1884 });
1885 };
1886 };
1887}
1888```
1889
1890### How to override a Python package for all Python versions using extensions? {#how-to-override-a-python-package-for-all-python-versions-using-extensions}
1891
1892The following overlay overrides the call to `buildPythonPackage` for the
1893`foo` package for all interpreters by appending a Python extension to the
1894`pythonPackagesExtensions` list of extensions.
1895
1896```nix
1897final: prev: {
1898 pythonPackagesExtensions = prev.pythonPackagesExtensions ++ [
1899 (
1900 python-final: python-prev: {
1901 foo = python-prev.foo.overridePythonAttrs (oldAttrs: {
1902 ...
1903 });
1904 }
1905 )
1906 ];
1907}
1908```
1909
1910### How to use Intel’s MKL with numpy and scipy? {#how-to-use-intels-mkl-with-numpy-and-scipy}
1911
1912MKL can be configured using an overlay. See the section "[Using overlays to
1913configure alternatives](#sec-overlays-alternatives-blas-lapack)".
1914
1915### What inputs do `setup_requires`, `install_requires` and `tests_require` map to? {#what-inputs-do-setup_requires-install_requires-and-tests_require-map-to}
1916
1917In a `setup.py` or `setup.cfg` it is common to declare dependencies:
1918
1919* `setup_requires` corresponds to `nativeBuildInputs`
1920* `install_requires` corresponds to `propagatedBuildInputs`
1921* `tests_require` corresponds to `nativeCheckInputs`
1922
1923## Contributing {#contributing}
1924
1925### Contributing guidelines {#contributing-guidelines}
1926
1927The following rules are desired to be respected:
1928
1929* Python libraries are called from `python-packages.nix` and packaged with
1930 `buildPythonPackage`. The expression of a library should be in
1931 `pkgs/development/python-modules/<name>/default.nix`.
1932* Python applications live outside of `python-packages.nix` and are packaged
1933 with `buildPythonApplication`.
1934* Make sure libraries build for all Python interpreters.
1935* By default we enable tests. Make sure the tests are found and, in the case of
1936 libraries, are passing for all interpreters. If certain tests fail they can be
1937 disabled individually. Try to avoid disabling the tests altogether. In any
1938 case, when you disable tests, leave a comment explaining why.
1939* Commit names of Python libraries should reflect that they are Python
1940 libraries, so write for example `pythonPackages.numpy: 1.11 -> 1.12`.
1941* Attribute names in `python-packages.nix` as well as `pname`s should match the
1942 library's name on PyPI, but be normalized according to [PEP
1943 0503](https://www.python.org/dev/peps/pep-0503/#normalized-names). This means
1944 that characters should be converted to lowercase and `.` and `_` should be
1945 replaced by a single `-` (foo-bar-baz instead of Foo__Bar.baz).
1946 If necessary, `pname` has to be given a different value within `fetchPypi`.
1947* Packages from sources such as GitHub and GitLab that do not exist on PyPI
1948 should not use a name that is already used on PyPI. When possible, they should
1949 use the package repository name prefixed with the owner (e.g. organization) name
1950 and using a `-` as delimiter.
1951* Attribute names in `python-packages.nix` should be sorted alphanumerically to
1952 avoid merge conflicts and ease locating attributes.
1953
1954## Package set maintenance {#python-package-set-maintenance}
1955
1956The whole Python package set has a lot of packages that do not see regular
1957updates, because they either are a very fragile component in the Python
1958ecosystem, like for example the `hypothesis` package, or packages that have
1959no maintainer, so maintenance falls back to the package set maintainers.
1960
1961### Updating packages in bulk {#python-package-bulk-updates}
1962
1963There is a tool to update alot of python libraries in bulk, it exists at
1964`maintainers/scripts/update-python-libraries` with this repository.
1965
1966It can quickly update minor or major versions for all packages selected
1967and create update commits, and supports the `fetchPypi`, `fetchurl` and
1968`fetchFromGitHub` fetchers. When updating lots of packages that are
1969hosted on GitHub, exporting a `GITHUB_API_TOKEN` is highly recommended.
1970
1971Updating packages in bulk leads to lots of breakages, which is why a
1972stabilization period on the `python-unstable` branch is required.
1973
1974If a package is fragile and often breaks during these bulks updates, it
1975may be reasonable to set `passthru.skipBulkUpdate = true` in the
1976derivation. This decision should not be made on a whim and should
1977always be supported by a qualifying comment.
1978
1979Once the branch is sufficiently stable it should normally be merged
1980into the `staging` branch.
1981
1982An exemplary call to update all python libraries between minor versions
1983would be:
1984
1985```ShellSession
1986$ maintainers/scripts/update-python-libraries --target minor --commit --use-pkgs-prefix pkgs/development/python-modules/**/default.nix
1987```
1988
1989## CPython Update Schedule {#python-cpython-update-schedule}
1990
1991With [PEP 602](https://www.python.org/dev/peps/pep-0602/), CPython now
1992follows a yearly release cadence. In nixpkgs, all supported interpreters
1993are made available, but only the most recent two
1994interpreters package sets are built; this is a compromise between being
1995the latest interpreter, and what the majority of the Python packages support.
1996
1997New CPython interpreters are released in October. Generally, it takes some
1998time for the majority of active Python projects to support the latest stable
1999interpreter. To help ease the migration for Nixpkgs users
2000between Python interpreters the schedule below will be used:
2001
2002| When | Event |
2003| --- | --- |
2004| After YY.11 Release | Bump CPython package set window. The latest and previous latest stable should now be built. |
2005| After YY.05 Release | Bump default CPython interpreter to latest stable. |
2006
2007In practice, this means that the Python community will have had a stable interpreter
2008for ~2 months before attempting to update the package set. And this will
2009allow for ~7 months for Python applications to support the latest interpreter.