pyrox.dev
1# Python {#python}
2
3## User Guide {#user-guide}
4
5### Using Python {#using-python}
6
7#### Overview {#overview}
8
9Several versions of the Python interpreter are available on Nix, as well as a
10high amount of packages. The attribute `python3` refers to the default
11interpreter, which is currently CPython 3.10. The attribute `python` refers to
12CPython 2.7 for backwards-compatibility. It is also possible to refer to
13specific versions, e.g. `python39` refers to CPython 3.9, and `pypy` refers to
14the default PyPy interpreter.
15
16Python is used a lot, and in different ways. This affects also how it is
17packaged. In the case of Python on Nix, an important distinction is made between
18whether the package is considered primarily an application, or whether it should
19be used as a library, i.e., of primary interest are the modules in
20`site-packages` that should be importable.
21
22In the Nixpkgs tree Python applications can be found throughout, depending on
23what they do, and are called from the main package set. Python libraries,
24however, are in separate sets, with one set per interpreter version.
25
26The interpreters have several common attributes. One of these attributes is
27`pkgs`, which is a package set of Python libraries for this specific
28interpreter. E.g., the `toolz` package corresponding to the default interpreter
29is `python.pkgs.toolz`, and the CPython 3.9 version is `python39.pkgs.toolz`.
30The main package set contains aliases to these package sets, e.g.
31`pythonPackages` refers to `python.pkgs` and `python39Packages` to
32`python39.pkgs`.
33
34#### Installing Python and packages {#installing-python-and-packages}
35
36The Nix and NixOS manuals explain how packages are generally installed. In the
37case of Python and Nix, it is important to make a distinction between whether the
38package is considered an application or a library.
39
40Applications on Nix are typically installed into your user profile imperatively
41using `nix-env -i`, and on NixOS declaratively by adding the package name to
42`environment.systemPackages` in `/etc/nixos/configuration.nix`. Dependencies
43such as libraries are automatically installed and should not be installed
44explicitly.
45
46The same goes for Python applications. Python applications can be installed in
47your profile, and will be wrapped to find their exact library dependencies,
48without impacting other applications or polluting your user environment.
49
50But Python libraries you would like to use for development cannot be installed,
51at least not individually, because they won't be able to find each other
52resulting in import errors. Instead, it is possible to create an environment
53with `python.buildEnv` or `python.withPackages` where the interpreter and other
54executables are wrapped to be able to find each other and all of the modules.
55
56In the following examples we will start by creating a simple, ad-hoc environment
57with a nix-shell that has `numpy` and `toolz` in Python 3.9; then we will create
58a re-usable environment in a single-file Python script; then we will create a
59full Python environment for development with this same environment.
60
61Philosphically, this should be familiar to users who are used to a `venv` style
62of development: individual projects create their own Python environments without
63impacting the global environment or each other.
64
65#### Ad-hoc temporary Python environment with `nix-shell` {#ad-hoc-temporary-python-environment-with-nix-shell}
66
67The simplest way to start playing with the way nix wraps and sets up Python
68environments is with `nix-shell` at the cmdline. These environments create a
69temporary shell session with a Python and a *precise* list of packages (plus
70their runtime dependencies), with no other Python packages in the Python
71interpreter's scope.
72
73To create a Python 3.9 session with `numpy` and `toolz` available, run:
74
75```sh
76$ nix-shell -p 'python39.withPackages(ps: with ps; [ numpy toolz ])'
77```
78
79By default `nix-shell` will start a `bash` session with this interpreter in our
80`PATH`, so if we then run:
81
82```Python console
83[nix-shell:~/src/nixpkgs]$ python3
84Python 3.9.12 (main, Mar 23 2022, 21:36:19)
85[GCC 11.3.0] on linux
86Type "help", "copyright", "credits" or "license" for more information.
87>>> import numpy; import toolz
88```
89
90Note that no other modules are in scope, even if they were imperatively
91installed into our user environment as a dependency of a Python application:
92
93```Python console
94>>> import requests
95Traceback (most recent call last):
96 File "<stdin>", line 1, in <module>
97ModuleNotFoundError: No module named 'requests'
98```
99
100We can add as many additional modules onto the `nix-shell` as we need, and we
101will still get 1 wrapped Python interpreter. We can start the interpreter
102directly like so:
103
104```sh
105$ nix-shell -p "python39.withPackages (ps: with ps; [ numpy toolz requests ])" --run python3
106this derivation will be built:
107 /nix/store/mpn7k6bkjl41fm51342rafaqfsl10qs4-python3-3.9.12-env.drv
108this path will be fetched (0.09 MiB download, 0.41 MiB unpacked):
109 /nix/store/5gaiacnzi096b6prc6aa1pwrhncmhc8b-python3.9-toolz-0.11.2
110copying path '/nix/store/5gaiacnzi096b6prc6aa1pwrhncmhc8b-python3.9-toolz-0.11.2' from 'https://cache.nixos.org'...
111building '/nix/store/mpn7k6bkjl41fm51342rafaqfsl10qs4-python3-3.9.12-env.drv'...
112created 279 symlinks in user environment
113Python 3.9.12 (main, Mar 23 2022, 21:36:19)
114[GCC 11.3.0] on linux
115Type "help", "copyright", "credits" or "license" for more information.
116>>> import requests
117>>>
118```
119
120Notice that this time it built a new Python environment, which now includes
121`requests`. Building an environment just creates wrapper scripts that expose the
122selected dependencies to the interpreter while re-using the actual modules. This
123means if any other env has installed `requests` or `numpy` in a different
124context, we don't need to recompile them -- we just recompile the wrapper script
125that sets up an interpreter pointing to them. This matters much more for "big"
126modules like `pytorch` or `tensorflow`.
127
128Module names usually match their names on [pypi.org](https://pypi.org/), but
129you can use the [Nixpkgs search website](https://nixos.org/nixos/packages.html)
130to find them as well (along with non-python packages).
131
132At this point we can create throwaway experimental Python environments with
133arbitrary dependencies. This is a good way to get a feel for how the Python
134interpreter and dependencies work in Nix and NixOS, but to do some actual
135development, we'll want to make it a bit more persistent.
136
137##### Running Python scripts and using `nix-shell` as shebang {#running-python-scripts-and-using-nix-shell-as-shebang}
138
139Sometimes, we have a script whose header looks like this:
140
141```python
142#!/usr/bin/env python3
143import numpy as np
144a = np.array([1,2])
145b = np.array([3,4])
146print(f"The dot product of {a} and {b} is: {np.dot(a, b)}")
147```
148
149Executing this script requires a `python3` that has `numpy`. Using what we learned
150in the previous section, we could startup a shell and just run it like so:
151
152```ShellSession
153$ nix-shell -p 'python39.withPackages(ps: with ps; [ numpy ])' --run 'python3 foo.py'
154The dot product of [1 2] and [3 4] is: 11
155```
156
157But if we maintain the script ourselves, and if there are more dependencies, it
158may be nice to encode those dependencies in source to make the script re-usable
159without that bit of knowledge. That can be done by using `nix-shell` as a
160[shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)), like so:
161
162```python
163#!/usr/bin/env nix-shell
164#!nix-shell -i python3 -p "python3.withPackages(ps: [ ps.numpy ])"
165import numpy as np
166a = np.array([1,2])
167b = np.array([3,4])
168print(f"The dot product of {a} and {b} is: {np.dot(a, b)}")
169```
170
171Then we simply execute it, without requiring any environment setup at all!
172
173```sh
174$ ./foo.py
175The dot product of [1 2] and [3 4] is: 11
176```
177
178If the dependencies are not available on the host where `foo.py` is executed, it
179will build or download them from a Nix binary cache prior to starting up, prior
180that it is executed on a machine with a multi-user nix installation.
181
182This provides a way to ship a self bootstrapping Python script, akin to a
183statically linked binary, where it can be run on any machine (provided nix is
184installed) without having to assume that `numpy` is installed globally on the
185system.
186
187By default it is pulling the import checkout of Nixpkgs itself from our nix
188channel, which is nice as it cache aligns with our other package builds, but we
189can make it fully reproducible by pinning the `nixpkgs` import:
190
191```python
192#!/usr/bin/env nix-shell
193#!nix-shell -i python3 -p "python3.withPackages(ps: [ ps.numpy ])"
194#!nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/d373d80b1207d52621961b16aa4a3438e4f98167.tar.gz
195import numpy as np
196a = np.array([1,2])
197b = np.array([3,4])
198print(f"The dot product of {a} and {b} is: {np.dot(a, b)}")
199```
200
201This will execute with the exact same versions of Python 3.8, numpy, and system
202dependencies a year from now as it does today, because it will always use
203exactly git commit `d373d80b1207d52621961b16aa4a3438e4f98167` of Nixpkgs for all
204of the package versions.
205
206This is also a great way to ensure the script executes identically on different
207servers.
208
209##### Load environment from `.nix` expression {#load-environment-from-.nix-expression}
210
211We've now seen how to create an ad-hoc temporary shell session, and how to
212create a single script with Python dependencies, but in the course of normal
213development we're usually working in an entire package repository.
214
215As explained in the Nix manual, `nix-shell` can also load an expression from a
216`.nix` file. Say we want to have Python 3.9, `numpy` and `toolz`, like before,
217in an environment. We can add a `shell.nix` file describing our dependencies:
218
219```nix
220with import <nixpkgs> {};
221(python39.withPackages (ps: [ps.numpy ps.toolz])).env
222```
223
224And then at the command line, just typing `nix-shell` produces the same
225environment as before. In a normal project, we'll likely have many more
226dependencies; this can provide a way for developers to share the environments
227with each other and with CI builders.
228
229What's happening here?
230
2311. We begin with importing the Nix Packages collections. `import <nixpkgs>`
232 imports the `<nixpkgs>` function, `{}` calls it and the `with` statement
233 brings all attributes of `nixpkgs` in the local scope. These attributes form
234 the main package set.
2352. Then we create a Python 3.9 environment with the `withPackages` function, as before.
2363. The `withPackages` function expects us to provide a function as an argument
237 that takes the set of all Python packages and returns a list of packages to
238 include in the environment. Here, we select the packages `numpy` and `toolz`
239 from the package set.
240
241To combine this with `mkShell` you can:
242
243```nix
244with import <nixpkgs> {};
245let
246 pythonEnv = python39.withPackages (ps: [
247 ps.numpy
248 ps.toolz
249 ]);
250in mkShell {
251 packages = [
252 pythonEnv
253
254 black
255 mypy
256
257 libffi
258 openssl
259 ];
260}
261```
262
263This will create a unified environment that has not just our Python interpreter
264and its Python dependencies, but also tools like `black` or `mypy` and libraries
265like `libffi` the `openssl` in scope. This is generic and can span any number of
266tools or languages across the Nixpkgs ecosystem.
267
268##### Installing environments globally on the system {#installing-environments-globally-on-the-system}
269
270Up to now, we've been creating environments scoped to an ad-hoc shell session,
271or a single script, or a single project. This is generally advisable, as it
272avoids pollution across contexts.
273
274However, sometimes we know we will often want a Python with some basic packages,
275and want this available without having to enter into a shell or build context.
276This can be useful to have things like vim/emacs editors and plugins or shell
277tools "just work" without having to set them up, or when running other software
278that expects packages to be installed globally.
279
280To create your own custom environment, create a file in `~/.config/nixpkgs/overlays/`
281that looks like this:
282
283```nix
284# ~/.config/nixpkgs/overlays/myEnv.nix
285self: super: {
286 myEnv = super.buildEnv {
287 name = "myEnv";
288 paths = [
289 # A Python 3 interpreter with some packages
290 (self.python3.withPackages (
291 ps: with ps; [
292 pyflakes
293 pytest
294 black
295 ]
296 ))
297
298 # Some other packages we'd like as part of this env
299 self.mypy
300 self.black
301 self.ripgrep
302 self.tmux
303 ];
304 };
305}
306```
307
308You can then build and install this to your profile with:
309
310```sh
311nix-env -iA myEnv
312```
313
314One limitation of this is that you can only have 1 Python env installed
315globally, since they conflict on the `python` to load out of your `PATH`.
316
317If you get a conflict or prefer to keep the setup clean, you can have `nix-env`
318atomically *uninstall* all other imperatively installed packages and replace
319your profile with just `myEnv` by using the `--replace` flag.
320
321##### Environment defined in `/etc/nixos/configuration.nix` {#environment-defined-in-etcnixosconfiguration.nix}
322
323For the sake of completeness, here's how to install the environment system-wide
324on NixOS.
325
326```nix
327{ # ...
328
329 environment.systemPackages = with pkgs; [
330 (python38.withPackages(ps: with ps; [ numpy toolz ]))
331 ];
332}
333```
334
335### Developing with Python {#developing-with-python}
336
337Above, we were mostly just focused on use cases and what to do to get started
338creating working Python environments in nix.
339
340Now that you know the basics to be up and running, it is time to take a step
341back and take a deeper look at how Python packages are packaged on Nix. Then,
342we will look at how you can use development mode with your code.
343
344#### Python library packages in Nixpkgs {#python-library-packages-in-nixpkgs}
345
346With Nix all packages are built by functions. The main function in Nix for
347building Python libraries is `buildPythonPackage`. Let's see how we can build the
348`toolz` package.
349
350```nix
351{ lib, buildPythonPackage, fetchPypi }:
352
353buildPythonPackage rec {
354 pname = "toolz";
355 version = "0.10.0";
356
357 src = fetchPypi {
358 inherit pname version;
359 sha256 = "08fdd5ef7c96480ad11c12d472de21acd32359996f69a5259299b540feba4560";
360 };
361
362 doCheck = false;
363
364 meta = with lib; {
365 homepage = "https://github.com/pytoolz/toolz";
366 description = "List processing tools and functional utilities";
367 license = licenses.bsd3;
368 maintainers = with maintainers; [ fridh ];
369 };
370}
371```
372
373What happens here? The function `buildPythonPackage` is called and as argument
374it accepts a set. In this case the set is a recursive set, `rec`. One of the
375arguments is the name of the package, which consists of a basename (generally
376following the name on PyPi) and a version. Another argument, `src` specifies the
377source, which in this case is fetched from PyPI using the helper function
378`fetchPypi`. The argument `doCheck` is used to set whether tests should be run
379when building the package. Furthermore, we specify some (optional) meta
380information. The output of the function is a derivation.
381
382An expression for `toolz` can be found in the Nixpkgs repository. As explained
383in the introduction of this Python section, a derivation of `toolz` is available
384for each interpreter version, e.g. `python39.pkgs.toolz` refers to the `toolz`
385derivation corresponding to the CPython 3.9 interpreter.
386
387The above example works when you're directly working on
388`pkgs/top-level/python-packages.nix` in the Nixpkgs repository. Often though,
389you will want to test a Nix expression outside of the Nixpkgs tree.
390
391The following expression creates a derivation for the `toolz` package,
392and adds it along with a `numpy` package to a Python environment.
393
394```nix
395with import <nixpkgs> {};
396
397( let
398 my_toolz = python39.pkgs.buildPythonPackage rec {
399 pname = "toolz";
400 version = "0.10.0";
401
402 src = python39.pkgs.fetchPypi {
403 inherit pname version;
404 sha256 = "08fdd5ef7c96480ad11c12d472de21acd32359996f69a5259299b540feba4560";
405 };
406
407 doCheck = false;
408
409 meta = {
410 homepage = "https://github.com/pytoolz/toolz/";
411 description = "List processing tools and functional utilities";
412 };
413 };
414
415 in python38.withPackages (ps: [ps.numpy my_toolz])
416).env
417```
418
419Executing `nix-shell` will result in an environment in which you can use
420Python 3.9 and the `toolz` package. As you can see we had to explicitly mention
421for which Python version we want to build a package.
422
423So, what did we do here? Well, we took the Nix expression that we used earlier
424to build a Python environment, and said that we wanted to include our own
425version of `toolz`, named `my_toolz`. To introduce our own package in the scope
426of `withPackages` we used a `let` expression. You can see that we used
427`ps.numpy` to select numpy from the nixpkgs package set (`ps`). We did not take
428`toolz` from the Nixpkgs package set this time, but instead took our own version
429that we introduced with the `let` expression.
430
431#### Handling dependencies {#handling-dependencies}
432
433Our example, `toolz`, does not have any dependencies on other Python packages or
434system libraries. According to the manual, `buildPythonPackage` uses the
435arguments `buildInputs` and `propagatedBuildInputs` to specify dependencies. If
436something is exclusively a build-time dependency, then the dependency should be
437included in `buildInputs`, but if it is (also) a runtime dependency, then it
438should be added to `propagatedBuildInputs`. Test dependencies are considered
439build-time dependencies and passed to `checkInputs`.
440
441The following example shows which arguments are given to `buildPythonPackage` in
442order to build [`datashape`](https://github.com/blaze/datashape).
443
444```nix
445{ lib, buildPythonPackage, fetchPypi, numpy, multipledispatch, python-dateutil, pytest }:
446
447buildPythonPackage rec {
448 pname = "datashape";
449 version = "0.4.7";
450
451 src = fetchPypi {
452 inherit pname version;
453 sha256 = "14b2ef766d4c9652ab813182e866f493475e65e558bed0822e38bf07bba1a278";
454 };
455
456 checkInputs = [ pytest ];
457 propagatedBuildInputs = [ numpy multipledispatch python-dateutil ];
458
459 meta = with lib; {
460 homepage = "https://github.com/ContinuumIO/datashape";
461 description = "A data description language";
462 license = licenses.bsd2;
463 maintainers = with maintainers; [ fridh ];
464 };
465}
466```
467
468We can see several runtime dependencies, `numpy`, `multipledispatch`, and
469`python-dateutil`. Furthermore, we have one `checkInputs`, i.e. `pytest`. `pytest` is a
470test runner and is only used during the `checkPhase` and is therefore not added
471to `propagatedBuildInputs`.
472
473In the previous case we had only dependencies on other Python packages to consider.
474Occasionally you have also system libraries to consider. E.g., `lxml` provides
475Python bindings to `libxml2` and `libxslt`. These libraries are only required
476when building the bindings and are therefore added as `buildInputs`.
477
478```nix
479{ lib, pkgs, buildPythonPackage, fetchPypi }:
480
481buildPythonPackage rec {
482 pname = "lxml";
483 version = "3.4.4";
484
485 src = fetchPypi {
486 inherit pname version;
487 sha256 = "16a0fa97hym9ysdk3rmqz32xdjqmy4w34ld3rm3jf5viqjx65lxk";
488 };
489
490 buildInputs = [ pkgs.libxml2 pkgs.libxslt ];
491
492 meta = with lib; {
493 description = "Pythonic binding for the libxml2 and libxslt libraries";
494 homepage = "https://lxml.de";
495 license = licenses.bsd3;
496 maintainers = with maintainers; [ sjourdois ];
497 };
498}
499```
500
501In this example `lxml` and Nix are able to work out exactly where the relevant
502files of the dependencies are. This is not always the case.
503
504The example below shows bindings to The Fastest Fourier Transform in the West,
505commonly known as FFTW. On Nix we have separate packages of FFTW for the
506different types of floats (`"single"`, `"double"`, `"long-double"`). The
507bindings need all three types, and therefore we add all three as `buildInputs`.
508The bindings don't expect to find each of them in a different folder, and
509therefore we have to set `LDFLAGS` and `CFLAGS`.
510
511```nix
512{ lib, pkgs, buildPythonPackage, fetchPypi, numpy, scipy }:
513
514buildPythonPackage rec {
515 pname = "pyFFTW";
516 version = "0.9.2";
517
518 src = fetchPypi {
519 inherit pname version;
520 sha256 = "f6bbb6afa93085409ab24885a1a3cdb8909f095a142f4d49e346f2bd1b789074";
521 };
522
523 buildInputs = [ pkgs.fftw pkgs.fftwFloat pkgs.fftwLongDouble];
524
525 propagatedBuildInputs = [ numpy scipy ];
526
527 # Tests cannot import pyfftw. pyfftw works fine though.
528 doCheck = false;
529
530 preConfigure = ''
531 export LDFLAGS="-L${pkgs.fftw.dev}/lib -L${pkgs.fftwFloat.out}/lib -L${pkgs.fftwLongDouble.out}/lib"
532 export CFLAGS="-I${pkgs.fftw.dev}/include -I${pkgs.fftwFloat.dev}/include -I${pkgs.fftwLongDouble.dev}/include"
533 '';
534
535 meta = with lib; {
536 description = "A pythonic wrapper around FFTW, the FFT library, presenting a unified interface for all the supported transforms";
537 homepage = "http://hgomersall.github.com/pyFFTW";
538 license = with licenses; [ bsd2 bsd3 ];
539 maintainers = with maintainers; [ fridh ];
540 };
541}
542```
543
544Note also the line `doCheck = false;`, we explicitly disabled running the test-suite.
545
546#### Testing Python Packages {#testing-python-packages}
547
548It is highly encouraged to have testing as part of the package build. This
549helps to avoid situations where the package was able to build and install,
550but is not usable at runtime. Currently, all packages will use the `test`
551command provided by the setup.py (i.e. `python setup.py test`). However,
552this is currently deprecated https://github.com/pypa/setuptools/pull/1878
553and your package should provide its own checkPhase.
554
555*NOTE:* The `checkPhase` for python maps to the `installCheckPhase` on a
556normal derivation. This is due to many python packages not behaving well
557to the pre-installed version of the package. Version info, and natively
558compiled extensions generally only exist in the install directory, and
559thus can cause issues when a test suite asserts on that behavior.
560
561*NOTE:* Tests should only be disabled if they don't agree with nix
562(e.g. external dependencies, network access, flakey tests), however,
563as many tests should be enabled as possible. Failing tests can still be
564a good indication that the package is not in a valid state.
565
566#### Using pytest {#using-pytest}
567
568Pytest is the most common test runner for python repositories. A trivial
569test run would be:
570
571```
572 checkInputs = [ pytest ];
573 checkPhase = "pytest";
574```
575
576However, many repositories' test suites do not translate well to nix's build
577sandbox, and will generally need many tests to be disabled.
578
579To filter tests using pytest, one can do the following:
580
581```
582 checkInputs = [ pytest ];
583 # avoid tests which need additional data or touch network
584 checkPhase = ''
585 pytest tests/ --ignore=tests/integration -k 'not download and not update'
586 '';
587```
588
589`--ignore` will tell pytest to ignore that file or directory from being
590collected as part of a test run. This is useful is a file uses a package
591which is not available in nixpkgs, thus skipping that test file is much
592easier than having to create a new package.
593
594`-k` is used to define a predicate for test names. In this example, we are
595filtering out tests which contain `download` or `update` in their test case name.
596Only one `-k` argument is allowed, and thus a long predicate should be concatenated
597with “\\” and wrapped to the next line.
598
599*NOTE:* In pytest==6.0.1, the use of “\\” to continue a line (e.g. `-k 'not download \'`) has
600been removed, in this case, it's recommended to use `pytestCheckHook`.
601
602#### Using pytestCheckHook {#using-pytestcheckhook}
603
604`pytestCheckHook` is a convenient hook which will substitute the setuptools
605`test` command for a `checkPhase` which runs `pytest`. This is also beneficial
606when a package may need many items disabled to run the test suite.
607
608Using the example above, the analagous `pytestCheckHook` usage would be:
609
610```
611 checkInputs = [ pytestCheckHook ];
612
613 # requires additional data
614 pytestFlagsArray = [ "tests/" "--ignore=tests/integration" ];
615
616 disabledTests = [
617 # touches network
618 "download"
619 "update"
620 ];
621
622 disabledTestPaths = [
623 "tests/test_failing.py"
624 ];
625```
626
627This is expecially useful when tests need to be conditionally disabled,
628for example:
629
630```
631 disabledTests = [
632 # touches network
633 "download"
634 "update"
635 ] ++ lib.optionals (pythonAtLeast "3.8") [
636 # broken due to python3.8 async changes
637 "async"
638 ] ++ lib.optionals stdenv.isDarwin [
639 # can fail when building with other packages
640 "socket"
641 ];
642```
643
644Trying to concatenate the related strings to disable tests in a regular
645`checkPhase` would be much harder to read. This also enables us to comment on
646why specific tests are disabled.
647
648#### Using pythonImportsCheck {#using-pythonimportscheck}
649
650Although unit tests are highly preferred to validate correctness of a package, not
651all packages have test suites that can be run easily, and some have none at all.
652To help ensure the package still works, `pythonImportsCheck` can attempt to import
653the listed modules.
654
655```
656 pythonImportsCheck = [ "requests" "urllib" ];
657```
658
659roughly translates to:
660
661```
662 postCheck = ''
663 PYTHONPATH=$out/${python.sitePackages}:$PYTHONPATH
664 python -c "import requests; import urllib"
665 '';
666```
667
668However, this is done in its own phase, and not dependent on whether `doCheck = true;`.
669
670This can also be useful in verifying that the package doesn't assume commonly
671present packages (e.g. `setuptools`).
672
673#### Using pythonRelaxDepsHook {#using-pythonrelaxdepshook}
674
675It is common for upstream to specify a range of versions for its package
676dependencies. This makes sense, since it ensures that the package will be built
677with a subset of packages that is well tested. However, this commonly causes
678issues when packaging in Nixpkgs, because the dependencies that this package
679may need are too new or old for the package to build correctly. We also cannot
680package multiple versions of the same package since this may cause conflicts
681in `PYTHONPATH`.
682
683One way to side step this issue is to relax the dependencies. This can be done
684by either removing the package version range or by removing the package
685declaration entirely. This can be done using the `pythonRelaxDepsHook` hook. For
686example, given the following `requirements.txt` file:
687
688```
689pkg1<1.0
690pkg2
691pkg3>=1.0,<=2.0
692```
693
694we can do:
695
696```
697 nativeBuildInputs = [ pythonRelaxDepsHook ];
698 pythonRelaxDeps = [ "pkg1" "pkg3" ];
699 pythonRemoveDeps = [ "pkg2" ];
700```
701
702which would result in the following `requirements.txt` file:
703
704```
705pkg1
706pkg3
707```
708
709Another option is to pass `true`, that will relax/remove all dependencies, for
710example:
711
712```
713 nativeBuildInputs = [ pythonRelaxDepsHook ];
714 pythonRelaxDeps = true;
715```
716
717which would result in the following `requirements.txt` file:
718
719```
720pkg1
721pkg2
722pkg3
723```
724
725In general you should always use `pythonRelaxDeps`, because `pythonRemoveDeps`
726will convert build errors into runtime errors. However `pythonRemoveDeps` may
727still be useful in exceptional cases, and also to remove dependencies wrongly
728declared by upstream (for example, declaring `black` as a runtime dependency
729instead of a dev dependency).
730
731Keep in mind that while the examples above are done with `requirements.txt`,
732`pythonRelaxDepsHook` works by modifying the resulting wheel file, so it should
733work in any of the formats supported by `buildPythonPackage` currently,
734with the exception of `other` (see `format` in
735[`buildPythonPackage` parameters](#buildpythonpackage-parameters) for more details).
736
737### Using unittestCheckHook {#using-unittestcheckhook}
738
739`unittestCheckHook` is a hook which will substitute the setuptools `test` command for a `checkPhase` which runs `python -m unittest discover`:
740
741```
742 checkInputs = [ unittestCheckHook ];
743
744 unittestFlags = [ "-s" "tests" "-v" ];
745```
746
747##### Using sphinxHook {#using-sphinxhook}
748
749The `sphinxHook` is a helpful tool to build documentation and manpages
750using the popular Sphinx documentation generator.
751It is setup to automatically find common documentation source paths and
752render them using the default `html` style.
753
754```
755 outputs = [
756 "out"
757 "doc"
758 ];
759
760 nativeBuildInputs = [
761 sphinxHook
762 ];
763```
764
765The hook will automatically build and install the artifact into the
766`doc` output, if it exists. It also provides an automatic diversion
767for the artifacts of the `man` builder into the `man` target.
768
769```
770 outputs = [
771 "out"
772 "doc"
773 "man"
774 ];
775
776 # Use multiple builders
777 sphinxBuilders = [
778 "singlehtml"
779 "man"
780 ];
781```
782
783Overwrite `sphinxRoot` when the hook is unable to find your
784documentation source root.
785
786```
787 # Configure sphinxRoot for uncommon paths
788 sphinxRoot = "weird/docs/path";
789```
790
791The hook is also available to packages outside the python ecosystem by
792referencing it using `sphinxHook` from top-level.
793
794### Develop local package {#develop-local-package}
795
796As a Python developer you're likely aware of [development mode](http://setuptools.readthedocs.io/en/latest/setuptools.html#development-mode)
797(`python setup.py develop`); instead of installing the package this command
798creates a special link to the project code. That way, you can run updated code
799without having to reinstall after each and every change you make. Development
800mode is also available. Let's see how you can use it.
801
802In the previous Nix expression the source was fetched from a url. We can also
803refer to a local source instead using `src = ./path/to/source/tree;`
804
805If we create a `shell.nix` file which calls `buildPythonPackage`, and if `src`
806is a local source, and if the local source has a `setup.py`, then development
807mode is activated.
808
809In the following example, we create a simple environment that has a Python 3.9
810version of our package in it, as well as its dependencies and other packages we
811like to have in the environment, all specified with `propagatedBuildInputs`.
812Indeed, we can just add any package we like to have in our environment to
813`propagatedBuildInputs`.
814
815```nix
816with import <nixpkgs> {};
817with python39Packages;
818
819buildPythonPackage rec {
820 name = "mypackage";
821 src = ./path/to/package/source;
822 propagatedBuildInputs = [ pytest numpy pkgs.libsndfile ];
823}
824```
825
826It is important to note that due to how development mode is implemented on Nix
827it is not possible to have multiple packages simultaneously in development mode.
828
829### Organising your packages {#organising-your-packages}
830
831So far we discussed how you can use Python on Nix, and how you can develop with
832it. We've looked at how you write expressions to package Python packages, and we
833looked at how you can create environments in which specified packages are
834available.
835
836At some point you'll likely have multiple packages which you would
837like to be able to use in different projects. In order to minimise unnecessary
838duplication we now look at how you can maintain a repository with your
839own packages. The important functions here are `import` and `callPackage`.
840
841### Including a derivation using `callPackage` {#including-a-derivation-using-callpackage}
842
843Earlier we created a Python environment using `withPackages`, and included the
844`toolz` package via a `let` expression.
845Let's split the package definition from the environment definition.
846
847We first create a function that builds `toolz` in `~/path/to/toolz/release.nix`
848
849```nix
850{ lib, buildPythonPackage }:
851
852buildPythonPackage rec {
853 pname = "toolz";
854 version = "0.10.0";
855
856 src = fetchPypi {
857 inherit pname version;
858 sha256 = "08fdd5ef7c96480ad11c12d472de21acd32359996f69a5259299b540feba4560";
859 };
860
861 meta = with lib; {
862 homepage = "https://github.com/pytoolz/toolz/";
863 description = "List processing tools and functional utilities";
864 license = licenses.bsd3;
865 maintainers = with maintainers; [ fridh ];
866 };
867}
868```
869
870It takes an argument `buildPythonPackage`. We now call this function using
871`callPackage` in the definition of our environment
872
873```nix
874with import <nixpkgs> {};
875
876( let
877 toolz = callPackage /path/to/toolz/release.nix {
878 buildPythonPackage = python38Packages.buildPythonPackage;
879 };
880 in python38.withPackages (ps: [ ps.numpy toolz ])
881).env
882```
883
884Important to remember is that the Python version for which the package is made
885depends on the `python` derivation that is passed to `buildPythonPackage`. Nix
886tries to automatically pass arguments when possible, which is why generally you
887don't explicitly define which `python` derivation should be used. In the above
888example we use `buildPythonPackage` that is part of the set `python38Packages`,
889and in this case the `python38` interpreter is automatically used.
890
891## Reference {#reference}
892
893### Interpreters {#interpreters}
894
895Versions 2.7, 3.7, 3.8, 3.9 and 3.10 of the CPython interpreter are available
896as respectively `python27`, `python37`, `python38`, `python39` and `python310`.
897The aliases `python2` and `python3` correspond to respectively `python27` and
898`python39`. The attribute `python` maps to `python2`. The PyPy interpreters
899compatible with Python 2.7 and 3 are available as `pypy27` and `pypy3`, with
900aliases `pypy2` mapping to `pypy27` and `pypy` mapping to `pypy2`. The Nix
901expressions for the interpreters can be found in
902`pkgs/development/interpreters/python`.
903
904All packages depending on any Python interpreter get appended
905`out/{python.sitePackages}` to `$PYTHONPATH` if such directory
906exists.
907
908#### Missing `tkinter` module standard library {#missing-tkinter-module-standard-library}
909
910To reduce closure size the `Tkinter`/`tkinter` is available as a separate package, `pythonPackages.tkinter`.
911
912#### Attributes on interpreters packages {#attributes-on-interpreters-packages}
913
914Each interpreter has the following attributes:
915
916- `libPrefix`. Name of the folder in `${python}/lib/` for corresponding interpreter.
917- `interpreter`. Alias for `${python}/bin/${executable}`.
918- `buildEnv`. Function to build python interpreter environments with extra packages bundled together. See section *python.buildEnv function* for usage and documentation.
919- `withPackages`. Simpler interface to `buildEnv`. See section *python.withPackages function* for usage and documentation.
920- `sitePackages`. Alias for `lib/${libPrefix}/site-packages`.
921- `executable`. Name of the interpreter executable, e.g. `python3.8`.
922- `pkgs`. Set of Python packages for that specific interpreter. The package set can be modified by overriding the interpreter and passing `packageOverrides`.
923
924### Optimizations {#optimizations}
925
926The Python interpreters are by default not built with optimizations enabled, because
927the builds are in that case not reproducible. To enable optimizations, override the
928interpreter of interest, e.g using
929
930```
931let
932 pkgs = import ./. {};
933 mypython = pkgs.python3.override {
934 enableOptimizations = true;
935 reproducibleBuild = false;
936 self = mypython;
937 };
938in mypython
939```
940
941### Building packages and applications {#building-packages-and-applications}
942
943Python libraries and applications that use `setuptools` or
944`distutils` are typically built with respectively the `buildPythonPackage` and
945`buildPythonApplication` functions. These two functions also support installing a `wheel`.
946
947All Python packages reside in `pkgs/top-level/python-packages.nix` and all
948applications elsewhere. In case a package is used as both a library and an
949application, then the package should be in `pkgs/top-level/python-packages.nix`
950since only those packages are made available for all interpreter versions. The
951preferred location for library expressions is in
952`pkgs/development/python-modules`. It is important that these packages are
953called from `pkgs/top-level/python-packages.nix` and not elsewhere, to guarantee
954the right version of the package is built.
955
956Based on the packages defined in `pkgs/top-level/python-packages.nix` an
957attribute set is created for each available Python interpreter. The available
958sets are
959
960* `pkgs.python27Packages`
961* `pkgs.python37Packages`
962* `pkgs.python38Packages`
963* `pkgs.python39Packages`
964* `pkgs.python310Packages`
965* `pkgs.python311Packages`
966* `pkgs.pypyPackages`
967
968and the aliases
969
970* `pkgs.python2Packages` pointing to `pkgs.python27Packages`
971* `pkgs.python3Packages` pointing to `pkgs.python39Packages`
972* `pkgs.pythonPackages` pointing to `pkgs.python2Packages`
973
974#### `buildPythonPackage` function {#buildpythonpackage-function}
975
976The `buildPythonPackage` function is implemented in
977`pkgs/development/interpreters/python/mk-python-derivation.nix`
978using setup hooks.
979
980The following is an example:
981
982```nix
983{ lib, buildPythonPackage, fetchPypi, hypothesis, setuptools-scm, attrs, py, setuptools, six, pluggy }:
984
985buildPythonPackage rec {
986 pname = "pytest";
987 version = "3.3.1";
988
989 src = fetchPypi {
990 inherit pname version;
991 sha256 = "cf8436dc59d8695346fcd3ab296de46425ecab00d64096cebe79fb51ecb2eb93";
992 };
993
994 postPatch = ''
995 # don't test bash builtins
996 rm testing/test_argcomplete.py
997 '';
998
999 checkInputs = [ hypothesis ];
1000 nativeBuildInputs = [ setuptools-scm ];
1001 propagatedBuildInputs = [ attrs py setuptools six pluggy ];
1002
1003 meta = with lib; {
1004 maintainers = with maintainers; [ domenkozar lovek323 madjar lsix ];
1005 description = "Framework for writing tests";
1006 };
1007}
1008```
1009
1010The `buildPythonPackage` mainly does four things:
1011
1012* In the `buildPhase`, it calls `${python.interpreter} setup.py bdist_wheel` to
1013 build a wheel binary zipfile.
1014* In the `installPhase`, it installs the wheel file using `pip install *.whl`.
1015* In the `postFixup` phase, the `wrapPythonPrograms` bash function is called to
1016 wrap all programs in the `$out/bin/*` directory to include `$PATH`
1017 environment variable and add dependent libraries to script's `sys.path`.
1018* In the `installCheck` phase, `${python.interpreter} setup.py test` is run.
1019
1020By default tests are run because `doCheck = true`. Test dependencies, like
1021e.g. the test runner, should be added to `checkInputs`.
1022
1023By default `meta.platforms` is set to the same value
1024as the interpreter unless overridden otherwise.
1025
1026##### `buildPythonPackage` parameters {#buildpythonpackage-parameters}
1027
1028All parameters from `stdenv.mkDerivation` function are still supported. The
1029following are specific to `buildPythonPackage`:
1030
1031* `catchConflicts ? true`: If `true`, abort package build if a package name
1032 appears more than once in dependency tree. Default is `true`.
1033* `disabled ? false`: If `true`, package is not built for the particular Python
1034 interpreter version.
1035* `dontWrapPythonPrograms ? false`: Skip wrapping of Python programs.
1036* `permitUserSite ? false`: Skip setting the `PYTHONNOUSERSITE` environment
1037 variable in wrapped programs.
1038* `format ? "setuptools"`: Format of the source. Valid options are
1039 `"setuptools"`, `"pyproject"`, `"flit"`, `"wheel"`, and `"other"`.
1040 `"setuptools"` is for when the source has a `setup.py` and `setuptools` is
1041 used to build a wheel, `flit`, in case `flit` should be used to build a wheel,
1042 and `wheel` in case a wheel is provided. Use `other` when a custom
1043 `buildPhase` and/or `installPhase` is needed.
1044* `makeWrapperArgs ? []`: A list of strings. Arguments to be passed to
1045 `makeWrapper`, which wraps generated binaries. By default, the arguments to
1046 `makeWrapper` set `PATH` and `PYTHONPATH` environment variables before calling
1047 the binary. Additional arguments here can allow a developer to set environment
1048 variables which will be available when the binary is run. For example,
1049 `makeWrapperArgs = ["--set FOO BAR" "--set BAZ QUX"]`.
1050* `namePrefix`: Prepends text to `${name}` parameter. In case of libraries, this
1051 defaults to `"python3.8-"` for Python 3.8, etc., and in case of applications
1052 to `""`.
1053* `pipInstallFlags ? []`: A list of strings. Arguments to be passed to `pip
1054 install`. To pass options to `python setup.py install`, use
1055 `--install-option`. E.g., `pipInstallFlags=["--install-option='--cpp_implementation'"]`.
1056* `pythonPath ? []`: List of packages to be added into `$PYTHONPATH`. Packages
1057 in `pythonPath` are not propagated (contrary to `propagatedBuildInputs`).
1058* `preShellHook`: Hook to execute commands before `shellHook`.
1059* `postShellHook`: Hook to execute commands after `shellHook`.
1060* `removeBinByteCode ? true`: Remove bytecode from `/bin`. Bytecode is only
1061 created when the filenames end with `.py`.
1062* `setupPyGlobalFlags ? []`: List of flags passed to `setup.py` command.
1063* `setupPyBuildFlags ? []`: List of flags passed to `setup.py build_ext` command.
1064
1065The `stdenv.mkDerivation` function accepts various parameters for describing
1066build inputs (see "Specifying dependencies"). The following are of special
1067interest for Python packages, either because these are primarily used, or
1068because their behaviour is different:
1069
1070* `nativeBuildInputs ? []`: Build-time only dependencies. Typically executables
1071 as well as the items listed in `setup_requires`.
1072* `buildInputs ? []`: Build and/or run-time dependencies that need to be
1073 compiled for the host machine. Typically non-Python libraries which are being
1074 linked.
1075* `checkInputs ? []`: Dependencies needed for running the `checkPhase`. These
1076 are added to `nativeBuildInputs` when `doCheck = true`. Items listed in
1077 `tests_require` go here.
1078* `propagatedBuildInputs ? []`: Aside from propagating dependencies,
1079 `buildPythonPackage` also injects code into and wraps executables with the
1080 paths included in this list. Items listed in `install_requires` go here.
1081
1082##### Overriding Python packages {#overriding-python-packages}
1083
1084The `buildPythonPackage` function has a `overridePythonAttrs` method that can be
1085used to override the package. In the following example we create an environment
1086where we have the `blaze` package using an older version of `pandas`. We
1087override first the Python interpreter and pass `packageOverrides` which contains
1088the overrides for packages in the package set.
1089
1090```nix
1091with import <nixpkgs> {};
1092
1093(let
1094 python = let
1095 packageOverrides = self: super: {
1096 pandas = super.pandas.overridePythonAttrs(old: rec {
1097 version = "0.19.1";
1098 src = super.fetchPypi {
1099 pname = "pandas";
1100 inherit version;
1101 sha256 = "08blshqj9zj1wyjhhw3kl2vas75vhhicvv72flvf1z3jvapgw295";
1102 };
1103 });
1104 };
1105 in pkgs.python3.override {inherit packageOverrides; self = python;};
1106
1107in python.withPackages(ps: [ps.blaze])).env
1108```
1109
1110#### Optional extra dependencies
1111
1112Some packages define optional dependencies for additional features. With
1113`setuptools` this is called `extras_require` and `flit` calls it
1114`extras-require`, while PEP 621 calls these `optional-dependencies`. A
1115method for supporting this is by declaring the extras of a package in its
1116`passthru`, e.g. in case of the package `dask`
1117
1118```nix
1119passthru.optional-dependencies = {
1120 complete = [ distributed ];
1121};
1122```
1123
1124and letting the package requiring the extra add the list to its dependencies
1125
1126```nix
1127propagatedBuildInputs = [
1128 ...
1129] ++ dask.optional-dependencies.complete;
1130```
1131
1132Note this method is preferred over adding parameters to builders, as that can
1133result in packages depending on different variants and thereby causing
1134collisions.
1135
1136#### `buildPythonApplication` function {#buildpythonapplication-function}
1137
1138The `buildPythonApplication` function is practically the same as
1139`buildPythonPackage`. The main purpose of this function is to build a Python
1140package where one is interested only in the executables, and not importable
1141modules. For that reason, when adding this package to a `python.buildEnv`, the
1142modules won't be made available.
1143
1144Another difference is that `buildPythonPackage` by default prefixes the names of
1145the packages with the version of the interpreter. Because this is irrelevant for
1146applications, the prefix is omitted.
1147
1148When packaging a Python application with `buildPythonApplication`, it should be
1149called with `callPackage` and passed `python` or `pythonPackages` (possibly
1150specifying an interpreter version), like this:
1151
1152```nix
1153{ lib, python3 }:
1154
1155python3.pkgs.buildPythonApplication rec {
1156 pname = "luigi";
1157 version = "2.7.9";
1158
1159 src = python3.pkgs.fetchPypi {
1160 inherit pname version;
1161 sha256 = "035w8gqql36zlan0xjrzz9j4lh9hs0qrsgnbyw07qs7lnkvbdv9x";
1162 };
1163
1164 propagatedBuildInputs = with python3.pkgs; [ tornado python-daemon ];
1165
1166 meta = with lib; {
1167 ...
1168 };
1169}
1170```
1171
1172This is then added to `all-packages.nix` just as any other application would be.
1173
1174```nix
1175luigi = callPackage ../applications/networking/cluster/luigi { };
1176```
1177
1178Since the package is an application, a consumer doesn't need to care about
1179Python versions or modules, which is why they don't go in `pythonPackages`.
1180
1181#### `toPythonApplication` function {#topythonapplication-function}
1182
1183A distinction is made between applications and libraries, however, sometimes a
1184package is used as both. In this case the package is added as a library to
1185`python-packages.nix` and as an application to `all-packages.nix`. To reduce
1186duplication the `toPythonApplication` can be used to convert a library to an
1187application.
1188
1189The Nix expression shall use `buildPythonPackage` and be called from
1190`python-packages.nix`. A reference shall be created from `all-packages.nix` to
1191the attribute in `python-packages.nix`, and the `toPythonApplication` shall be
1192applied to the reference:
1193
1194```nix
1195youtube-dl = with pythonPackages; toPythonApplication youtube-dl;
1196```
1197
1198#### `toPythonModule` function {#topythonmodule-function}
1199
1200In some cases, such as bindings, a package is created using
1201`stdenv.mkDerivation` and added as attribute in `all-packages.nix`. The Python
1202bindings should be made available from `python-packages.nix`. The
1203`toPythonModule` function takes a derivation and makes certain Python-specific
1204modifications.
1205
1206```nix
1207opencv = toPythonModule (pkgs.opencv.override {
1208 enablePython = true;
1209 pythonPackages = self;
1210});
1211```
1212
1213Do pay attention to passing in the right Python version!
1214
1215#### `python.buildEnv` function {#python.buildenv-function}
1216
1217Python environments can be created using the low-level `pkgs.buildEnv` function.
1218This example shows how to create an environment that has the Pyramid Web Framework.
1219Saving the following as `default.nix`
1220
1221```nix
1222with import <nixpkgs> {};
1223
1224python.buildEnv.override {
1225 extraLibs = [ pythonPackages.pyramid ];
1226 ignoreCollisions = true;
1227}
1228```
1229
1230and running `nix-build` will create
1231
1232```
1233/nix/store/cf1xhjwzmdki7fasgr4kz6di72ykicl5-python-2.7.8-env
1234```
1235
1236with wrapped binaries in `bin/`.
1237
1238You can also use the `env` attribute to create local environments with needed
1239packages installed. This is somewhat comparable to `virtualenv`. For example,
1240running `nix-shell` with the following `shell.nix`
1241
1242```nix
1243with import <nixpkgs> {};
1244
1245(python3.buildEnv.override {
1246 extraLibs = with python3Packages; [ numpy requests ];
1247}).env
1248```
1249
1250will drop you into a shell where Python will have the
1251specified packages in its path.
1252
1253##### `python.buildEnv` arguments {#python.buildenv-arguments}
1254
1255
1256* `extraLibs`: List of packages installed inside the environment.
1257* `postBuild`: Shell command executed after the build of environment.
1258* `ignoreCollisions`: Ignore file collisions inside the environment (default is `false`).
1259* `permitUserSite`: Skip setting the `PYTHONNOUSERSITE` environment variable in
1260 wrapped binaries in the environment.
1261
1262#### `python.withPackages` function {#python.withpackages-function}
1263
1264The `python.withPackages` function provides a simpler interface to the `python.buildEnv` functionality.
1265It takes a function as an argument that is passed the set of python packages and returns the list
1266of the packages to be included in the environment. Using the `withPackages` function, the previous
1267example for the Pyramid Web Framework environment can be written like this:
1268
1269```nix
1270with import <nixpkgs> {};
1271
1272python.withPackages (ps: [ps.pyramid])
1273```
1274
1275`withPackages` passes the correct package set for the specific interpreter
1276version as an argument to the function. In the above example, `ps` equals
1277`pythonPackages`. But you can also easily switch to using python3:
1278
1279```nix
1280with import <nixpkgs> {};
1281
1282python3.withPackages (ps: [ps.pyramid])
1283```
1284
1285Now, `ps` is set to `python3Packages`, matching the version of the interpreter.
1286
1287As `python.withPackages` simply uses `python.buildEnv` under the hood, it also
1288supports the `env` attribute. The `shell.nix` file from the previous section can
1289thus be also written like this:
1290
1291```nix
1292with import <nixpkgs> {};
1293
1294(python38.withPackages (ps: [ps.numpy ps.requests])).env
1295```
1296
1297In contrast to `python.buildEnv`, `python.withPackages` does not support the
1298more advanced options such as `ignoreCollisions = true` or `postBuild`. If you
1299need them, you have to use `python.buildEnv`.
1300
1301Python 2 namespace packages may provide `__init__.py` that collide. In that case
1302`python.buildEnv` should be used with `ignoreCollisions = true`.
1303
1304#### Setup hooks {#setup-hooks}
1305
1306The following are setup hooks specifically for Python packages. Most of these
1307are used in `buildPythonPackage`.
1308
1309- `eggUnpackhook` to move an egg to the correct folder so it can be installed
1310 with the `eggInstallHook`
1311- `eggBuildHook` to skip building for eggs.
1312- `eggInstallHook` to install eggs.
1313- `flitBuildHook` to build a wheel using `flit`.
1314- `pipBuildHook` to build a wheel using `pip` and PEP 517. Note a build system
1315 (e.g. `setuptools` or `flit`) should still be added as `nativeBuildInput`.
1316- `pipInstallHook` to install wheels.
1317- `pytestCheckHook` to run tests with `pytest`. See [example usage](#using-pytestcheckhook).
1318- `pythonCatchConflictsHook` to check whether a Python package is not already existing.
1319- `pythonImportsCheckHook` to check whether importing the listed modules works.
1320- `pythonRelaxDepsHook` will relax Python dependencies restrictions for the package.
1321 See [example usage](#using-pythonrelaxdepshook).
1322- `pythonRemoveBinBytecode` to remove bytecode from the `/bin` folder.
1323- `setuptoolsBuildHook` to build a wheel using `setuptools`.
1324- `setuptoolsCheckHook` to run tests with `python setup.py test`.
1325- `sphinxHook` to build documentation and manpages using Sphinx.
1326- `venvShellHook` to source a Python 3 `venv` at the `venvDir` location. A
1327 `venv` is created if it does not yet exist. `postVenvCreation` can be used to
1328 to run commands only after venv is first created.
1329- `wheelUnpackHook` to move a wheel to the correct folder so it can be installed
1330 with the `pipInstallHook`.
1331- `unittestCheckHook` will run tests with `python -m unittest discover`. See [example usage](#using-unittestcheckhook).
1332
1333### Development mode {#development-mode}
1334
1335Development or editable mode is supported. To develop Python packages
1336`buildPythonPackage` has additional logic inside `shellPhase` to run `pip
1337install -e . --prefix $TMPDIR/`for the package.
1338
1339Warning: `shellPhase` is executed only if `setup.py` exists.
1340
1341Given a `default.nix`:
1342
1343```nix
1344with import <nixpkgs> {};
1345
1346pythonPackages.buildPythonPackage {
1347 name = "myproject";
1348 buildInputs = with pythonPackages; [ pyramid ];
1349
1350 src = ./.;
1351}
1352```
1353
1354Running `nix-shell` with no arguments should give you the environment in which
1355the package would be built with `nix-build`.
1356
1357Shortcut to setup environments with C headers/libraries and Python packages:
1358
1359```shell
1360nix-shell -p pythonPackages.pyramid zlib libjpeg git
1361```
1362
1363Note: There is a boolean value `lib.inNixShell` set to `true` if nix-shell is invoked.
1364
1365### Tools {#tools}
1366
1367Packages inside nixpkgs are written by hand. However many tools exist in
1368community to help save time. No tool is preferred at the moment.
1369
1370- [pypi2nix](https://github.com/nix-community/pypi2nix): Generate Nix
1371 expressions for your Python project. Note that [sharing derivations from
1372 pypi2nix with nixpkgs is possible but not
1373 encouraged](https://github.com/nix-community/pypi2nix/issues/222#issuecomment-443497376).
1374- [nixpkgs-pytools](https://github.com/nix-community/nixpkgs-pytools)
1375- [poetry2nix](https://github.com/nix-community/poetry2nix)
1376
1377### Deterministic builds {#deterministic-builds}
1378
1379The Python interpreters are now built deterministically. Minor modifications had
1380to be made to the interpreters in order to generate deterministic bytecode. This
1381has security implications and is relevant for those using Python in a
1382`nix-shell`.
1383
1384When the environment variable `DETERMINISTIC_BUILD` is set, all bytecode will
1385have timestamp 1. The `buildPythonPackage` function sets `DETERMINISTIC_BUILD=1`
1386and [PYTHONHASHSEED=0](https://docs.python.org/3.8/using/cmdline.html#envvar-PYTHONHASHSEED).
1387Both are also exported in `nix-shell`.
1388
1389### Automatic tests {#automatic-tests}
1390
1391It is recommended to test packages as part of the build process.
1392Source distributions (`sdist`) often include test files, but not always.
1393
1394By default the command `python setup.py test` is run as part of the
1395`checkPhase`, but often it is necessary to pass a custom `checkPhase`. An
1396example of such a situation is when `py.test` is used.
1397
1398#### Common issues {#common-issues}
1399
1400* Non-working tests can often be deselected. By default `buildPythonPackage`
1401 runs `python setup.py test`. Most Python modules follows the standard test
1402 protocol where the pytest runner can be used instead. `py.test` supports a
1403 `-k` parameter to ignore test methods or classes:
1404
1405 ```nix
1406 buildPythonPackage {
1407 # ...
1408 # assumes the tests are located in tests
1409 checkInputs = [ pytest ];
1410 checkPhase = ''
1411 py.test -k 'not function_name and not other_function' tests
1412 '';
1413 }
1414 ```
1415
1416* Tests that attempt to access `$HOME` can be fixed by using the following
1417 work-around before running tests (e.g. `preCheck`): `export HOME=$(mktemp -d)`
1418
1419## FAQ {#faq}
1420
1421### How to solve circular dependencies? {#how-to-solve-circular-dependencies}
1422
1423Consider the packages `A` and `B` that depend on each other. When packaging `B`,
1424a solution is to override package `A` not to depend on `B` as an input. The same
1425should also be done when packaging `A`.
1426
1427### How to override a Python package? {#how-to-override-a-python-package}
1428
1429We can override the interpreter and pass `packageOverrides`. In the following
1430example we rename the `pandas` package and build it.
1431
1432```nix
1433with import <nixpkgs> {};
1434
1435(let
1436 python = let
1437 packageOverrides = self: super: {
1438 pandas = super.pandas.overridePythonAttrs(old: {name="foo";});
1439 };
1440 in pkgs.python38.override {inherit packageOverrides;};
1441
1442in python.withPackages(ps: [ps.pandas])).env
1443```
1444
1445Using `nix-build` on this expression will build an environment that contains the
1446package `pandas` but with the new name `foo`.
1447
1448All packages in the package set will use the renamed package. A typical use case
1449is to switch to another version of a certain package. For example, in the
1450Nixpkgs repository we have multiple versions of `django` and `scipy`. In the
1451following example we use a different version of `scipy` and create an
1452environment that uses it. All packages in the Python package set will now use
1453the updated `scipy` version.
1454
1455```nix
1456with import <nixpkgs> {};
1457
1458( let
1459 packageOverrides = self: super: {
1460 scipy = super.scipy_0_17;
1461 };
1462 in (pkgs.python38.override {inherit packageOverrides;}).withPackages (ps: [ps.blaze])
1463).env
1464```
1465
1466The requested package `blaze` depends on `pandas` which itself depends on `scipy`.
1467
1468If you want the whole of Nixpkgs to use your modifications, then you can use
1469`overlays` as explained in this manual. In the following example we build a
1470`inkscape` using a different version of `numpy`.
1471
1472```nix
1473let
1474 pkgs = import <nixpkgs> {};
1475 newpkgs = import pkgs.path { overlays = [ (self: super: {
1476 python38 = let
1477 packageOverrides = python-self: python-super: {
1478 numpy = python-super.numpy_1_18;
1479 };
1480 in super.python38.override {inherit packageOverrides;};
1481 } ) ]; };
1482in newpkgs.inkscape
1483```
1484
1485### `python setup.py bdist_wheel` cannot create .whl {#python-setup.py-bdist_wheel-cannot-create-.whl}
1486
1487Executing `python setup.py bdist_wheel` in a `nix-shell`fails with
1488
1489```
1490ValueError: ZIP does not support timestamps before 1980
1491```
1492
1493This is because files from the Nix store (which have a timestamp of the UNIX
1494epoch of January 1, 1970) are included in the .ZIP, but .ZIP archives follow the
1495DOS convention of counting timestamps from 1980.
1496
1497The command `bdist_wheel` reads the `SOURCE_DATE_EPOCH` environment variable,
1498which `nix-shell` sets to 1. Unsetting this variable or giving it a value
1499corresponding to 1980 or later enables building wheels.
1500
1501Use 1980 as timestamp:
1502
1503```shell
1504nix-shell --run "SOURCE_DATE_EPOCH=315532800 python3 setup.py bdist_wheel"
1505```
1506
1507or the current time:
1508
1509```shell
1510nix-shell --run "SOURCE_DATE_EPOCH=$(date +%s) python3 setup.py bdist_wheel"
1511```
1512
1513or unset `SOURCE_DATE_EPOCH`:
1514
1515```shell
1516nix-shell --run "unset SOURCE_DATE_EPOCH; python3 setup.py bdist_wheel"
1517```
1518
1519### `install_data` / `data_files` problems {#install_data-data_files-problems}
1520
1521If you get the following error:
1522
1523```
1524could not create '/nix/store/6l1bvljpy8gazlsw2aw9skwwp4pmvyxw-python-2.7.8/etc':
1525Permission denied
1526```
1527
1528This is a [known bug](https://github.com/pypa/setuptools/issues/130) in
1529`setuptools`. Setuptools `install_data` does not respect `--prefix`. An example
1530of such package using the feature is `pkgs/tools/X11/xpra/default.nix`.
1531
1532As workaround install it as an extra `preInstall` step:
1533
1534```shell
1535${python.interpreter} setup.py install_data --install-dir=$out --root=$out
1536sed -i '/ = data\_files/d' setup.py
1537```
1538
1539### Rationale of non-existent global site-packages {#rationale-of-non-existent-global-site-packages}
1540
1541On most operating systems a global `site-packages` is maintained. This however
1542becomes problematic if you want to run multiple Python versions or have multiple
1543versions of certain libraries for your projects. Generally, you would solve such
1544issues by creating virtual environments using `virtualenv`.
1545
1546On Nix each package has an isolated dependency tree which, in the case of
1547Python, guarantees the right versions of the interpreter and libraries or
1548packages are available. There is therefore no need to maintain a global `site-packages`.
1549
1550If you want to create a Python environment for development, then the recommended
1551method is to use `nix-shell`, either with or without the `python.buildEnv`
1552function.
1553
1554### How to consume Python modules using pip in a virtual environment like I am used to on other Operating Systems? {#how-to-consume-python-modules-using-pip-in-a-virtual-environment-like-i-am-used-to-on-other-operating-systems}
1555
1556While this approach is not very idiomatic from Nix perspective, it can still be
1557useful when dealing with pre-existing projects or in situations where it's not
1558feasible or desired to write derivations for all required dependencies.
1559
1560This is an example of a `default.nix` for a `nix-shell`, which allows to consume
1561a virtual environment created by `venv`, and install Python modules through
1562`pip` the traditional way.
1563
1564Create this `default.nix` file, together with a `requirements.txt` and simply
1565execute `nix-shell`.
1566
1567```nix
1568with import <nixpkgs> { };
1569
1570let
1571 pythonPackages = python3Packages;
1572in pkgs.mkShell rec {
1573 name = "impurePythonEnv";
1574 venvDir = "./.venv";
1575 buildInputs = [
1576 # A Python interpreter including the 'venv' module is required to bootstrap
1577 # the environment.
1578 pythonPackages.python
1579
1580 # This executes some shell code to initialize a venv in $venvDir before
1581 # dropping into the shell
1582 pythonPackages.venvShellHook
1583
1584 # Those are dependencies that we would like to use from nixpkgs, which will
1585 # add them to PYTHONPATH and thus make them accessible from within the venv.
1586 pythonPackages.numpy
1587 pythonPackages.requests
1588
1589 # In this particular example, in order to compile any binary extensions they may
1590 # require, the Python modules listed in the hypothetical requirements.txt need
1591 # the following packages to be installed locally:
1592 taglib
1593 openssl
1594 git
1595 libxml2
1596 libxslt
1597 libzip
1598 zlib
1599 ];
1600
1601 # Run this command, only after creating the virtual environment
1602 postVenvCreation = ''
1603 unset SOURCE_DATE_EPOCH
1604 pip install -r requirements.txt
1605 '';
1606
1607 # Now we can execute any commands within the virtual environment.
1608 # This is optional and can be left out to run pip manually.
1609 postShellHook = ''
1610 # allow pip to install wheels
1611 unset SOURCE_DATE_EPOCH
1612 '';
1613
1614}
1615```
1616
1617In case the supplied venvShellHook is insufficient, or when Python 2 support is
1618needed, you can define your own shell hook and adapt to your needs like in the
1619following example:
1620
1621```nix
1622with import <nixpkgs> { };
1623
1624let
1625 venvDir = "./.venv";
1626 pythonPackages = python3Packages;
1627in pkgs.mkShell rec {
1628 name = "impurePythonEnv";
1629 buildInputs = [
1630 pythonPackages.python
1631 # Needed when using python 2.7
1632 # pythonPackages.virtualenv
1633 # ...
1634 ];
1635
1636 # This is very close to how venvShellHook is implemented, but
1637 # adapted to use 'virtualenv'
1638 shellHook = ''
1639 SOURCE_DATE_EPOCH=$(date +%s)
1640
1641 if [ -d "${venvDir}" ]; then
1642 echo "Skipping venv creation, '${venvDir}' already exists"
1643 else
1644 echo "Creating new venv environment in path: '${venvDir}'"
1645 # Note that the module venv was only introduced in python 3, so for 2.7
1646 # this needs to be replaced with a call to virtualenv
1647 ${pythonPackages.python.interpreter} -m venv "${venvDir}"
1648 fi
1649
1650 # Under some circumstances it might be necessary to add your virtual
1651 # environment to PYTHONPATH, which you can do here too;
1652 # PYTHONPATH=$PWD/${venvDir}/${pythonPackages.python.sitePackages}/:$PYTHONPATH
1653
1654 source "${venvDir}/bin/activate"
1655
1656 # As in the previous example, this is optional.
1657 pip install -r requirements.txt
1658 '';
1659}
1660```
1661
1662Note that the `pip install` is an imperative action. So every time `nix-shell`
1663is executed it will attempt to download the Python modules listed in
1664requirements.txt. However these will be cached locally within the `virtualenv`
1665folder and not downloaded again.
1666
1667### How to override a Python package from `configuration.nix`? {#how-to-override-a-python-package-from-configuration.nix}
1668
1669If you need to change a package's attribute(s) from `configuration.nix` you could do:
1670
1671```nix
1672 nixpkgs.config.packageOverrides = super: {
1673 python3 = super.python3.override {
1674 packageOverrides = python-self: python-super: {
1675 twisted = python-super.twisted.overridePythonAttrs (oldAttrs: {
1676 src = super.fetchPypi {
1677 pname = "twisted";
1678 version = "19.10.0";
1679 sha256 = "7394ba7f272ae722a74f3d969dcf599bc4ef093bc392038748a490f1724a515d";
1680 extension = "tar.bz2";
1681 };
1682 });
1683 };
1684 };
1685 };
1686```
1687
1688`pythonPackages.twisted` is now globally overridden.
1689All packages and also all NixOS services that reference `twisted`
1690(such as `services.buildbot-worker`) now use the new definition.
1691Note that `python-super` refers to the old package set and `python-self`
1692to the new, overridden version.
1693
1694To modify only a Python package set instead of a whole Python derivation, use
1695this snippet:
1696
1697```nix
1698 myPythonPackages = pythonPackages.override {
1699 overrides = self: super: {
1700 twisted = ...;
1701 };
1702 }
1703```
1704
1705### How to override a Python package using overlays? {#how-to-override-a-python-package-using-overlays}
1706
1707Use the following overlay template:
1708
1709```nix
1710self: super: {
1711 python = super.python.override {
1712 packageOverrides = python-self: python-super: {
1713 twisted = python-super.twisted.overrideAttrs (oldAttrs: {
1714 src = super.fetchPypi {
1715 pname = "twisted";
1716 version = "19.10.0";
1717 sha256 = "7394ba7f272ae722a74f3d969dcf599bc4ef093bc392038748a490f1724a515d";
1718 extension = "tar.bz2";
1719 };
1720 });
1721 };
1722 };
1723}
1724```
1725
1726### How to override a Python package for all Python versions using extensions? {#how-to-override-a-python-package-for-all-python-versions-using-extensions}
1727
1728The following overlay overrides the call to `buildPythonPackage` for the
1729`foo` package for all interpreters by appending a Python extension to the
1730`pythonPackagesExtensions` list of extensions.
1731
1732```nix
1733final: prev: {
1734 pythonPackagesExtensions = prev.pythonPackagesExtensions ++ [
1735 (
1736 python-final: python-prev: {
1737 foo = python-prev.foo.overridePythonAttrs (oldAttrs: {
1738 ...
1739 });
1740 }
1741 )
1742 ];
1743}
1744```
1745
1746### How to use Intel’s MKL with numpy and scipy? {#how-to-use-intels-mkl-with-numpy-and-scipy}
1747
1748MKL can be configured using an overlay. See the section "[Using overlays to
1749configure alternatives](#sec-overlays-alternatives-blas-lapack)".
1750
1751### What inputs do `setup_requires`, `install_requires` and `tests_require` map to? {#what-inputs-do-setup_requires-install_requires-and-tests_require-map-to}
1752
1753In a `setup.py` or `setup.cfg` it is common to declare dependencies:
1754
1755* `setup_requires` corresponds to `nativeBuildInputs`
1756* `install_requires` corresponds to `propagatedBuildInputs`
1757* `tests_require` corresponds to `checkInputs`
1758
1759## Contributing {#contributing}
1760
1761### Contributing guidelines {#contributing-guidelines}
1762
1763The following rules are desired to be respected:
1764
1765* Python libraries are called from `python-packages.nix` and packaged with
1766 `buildPythonPackage`. The expression of a library should be in
1767 `pkgs/development/python-modules/<name>/default.nix`.
1768* Python applications live outside of `python-packages.nix` and are packaged
1769 with `buildPythonApplication`.
1770* Make sure libraries build for all Python interpreters.
1771* By default we enable tests. Make sure the tests are found and, in the case of
1772 libraries, are passing for all interpreters. If certain tests fail they can be
1773 disabled individually. Try to avoid disabling the tests altogether. In any
1774 case, when you disable tests, leave a comment explaining why.
1775* Commit names of Python libraries should reflect that they are Python
1776 libraries, so write for example `pythonPackages.numpy: 1.11 -> 1.12`.
1777* Attribute names in `python-packages.nix` as well as `pname`s should match the
1778 library's name on PyPI, but be normalized according to [PEP
1779 0503](https://www.python.org/dev/peps/pep-0503/#normalized-names). This means
1780 that characters should be converted to lowercase and `.` and `_` should be
1781 replaced by a single `-` (foo-bar-baz instead of Foo__Bar.baz).
1782 If necessary, `pname` has to be given a different value within `fetchPypi`.
1783* Packages from sources such as GitHub and GitLab that do not exist on PyPI
1784 should not use a name that is already used on PyPI. When possible, they should
1785 use the package repository name prefixed with the owner (e.g. organization) name
1786 and using a `-` as delimiter.
1787* Attribute names in `python-packages.nix` should be sorted alphanumerically to
1788 avoid merge conflicts and ease locating attributes.
1789
1790## Package set maintenance
1791
1792The whole Python package set has a lot of packages that do not see regular
1793updates, because they either are a very fragile component in the Python
1794ecosystem, like for example the `hypothesis` package, or packages that have
1795no maintainer, so maintenance falls back to the package set maintainers.
1796
1797### Updating packages in bulk
1798
1799There is a tool to update alot of python libraries in bulk, it exists at
1800`maintainers/scripts/update-python-libraries` with this repository.
1801
1802It can quickly update minor or major versions for all packages selected
1803and create update commits, and supports the `fetchPypi`, `fetchurl` and
1804`fetchFromGitHub` fetchers. When updating lots of packages that are
1805hosted on GitHub, exporting a `GITHUB_API_TOKEN` is highly recommended.
1806
1807Updating packages in bulk leads to lots of breakages, which is why a
1808stabilization period on the `python-unstable` branch is required.
1809
1810Once the branch is sufficiently stable it should normally be merged
1811into the `staging` branch.
1812
1813An exemplary call to update all python libraries between minor versions
1814would be:
1815
1816```ShellSession
1817$ maintainers/scripts/update-python-libraries --target minor --commit --use-pkgs-prefix pkgs/development/python-modules/**/default.nix
1818```
1819
1820## CPython Update Schedule
1821
1822With [PEP 602](https://www.python.org/dev/peps/pep-0602/), CPython now
1823follows a yearly release cadence. In nixpkgs, all supported interpreters
1824are made available, but only the most recent two
1825interpreters package sets are built; this is a compromise between being
1826the latest interpreter, and what the majority of the Python packages support.
1827
1828New CPython interpreters are released in October. Generally, it takes some
1829time for the majority of active Python projects to support the latest stable
1830interpreter. To help ease the migration for Nixpkgs users
1831between Python interpreters the schedule below will be used:
1832
1833| When | Event |
1834| --- | --- |
1835| After YY.11 Release | Bump CPython package set window. The latest and previous latest stable should now be built. |
1836| After YY.05 Release | Bump default CPython interpreter to latest stable. |
1837
1838In practice, this means that the Python community will have had a stable interpreter
1839for ~2 months before attempting to update the package set. And this will
1840allow for ~7 months for Python applications to support the latest interpreter.