pyrox.dev
1# Python {#python}
2
3## User Guide {#user-guide}
4
5### Using Python {#using-python}
6
7#### Overview {#overview}
8
9Several versions of the Python interpreter are available on Nix, as well as a
10high amount of packages. The attribute `python3` refers to the default
11interpreter, which is currently CPython 3.9. The attribute `python` refers to
12CPython 2.7 for backwards-compatibility. It is also possible to refer to
13specific versions, e.g. `python38` refers to CPython 3.8, and `pypy` refers to
14the default PyPy interpreter.
15
16Python is used a lot, and in different ways. This affects also how it is
17packaged. In the case of Python on Nix, an important distinction is made between
18whether the package is considered primarily an application, or whether it should
19be used as a library, i.e., of primary interest are the modules in
20`site-packages` that should be importable.
21
22In the Nixpkgs tree Python applications can be found throughout, depending on
23what they do, and are called from the main package set. Python libraries,
24however, are in separate sets, with one set per interpreter version.
25
26The interpreters have several common attributes. One of these attributes is
27`pkgs`, which is a package set of Python libraries for this specific
28interpreter. E.g., the `toolz` package corresponding to the default interpreter
29is `python.pkgs.toolz`, and the CPython 3.8 version is `python38.pkgs.toolz`.
30The main package set contains aliases to these package sets, e.g.
31`pythonPackages` refers to `python.pkgs` and `python38Packages` to
32`python38.pkgs`.
33
34#### Installing Python and packages {#installing-python-and-packages}
35
36The Nix and NixOS manuals explain how packages are generally installed. In the
37case of Python and Nix, it is important to make a distinction between whether the
38package is considered an application or a library.
39
40Applications on Nix are typically installed into your user profile imperatively
41using `nix-env -i`, and on NixOS declaratively by adding the package name to
42`environment.systemPackages` in `/etc/nixos/configuration.nix`. Dependencies
43such as libraries are automatically installed and should not be installed
44explicitly.
45
46The same goes for Python applications. Python applications can be installed in
47your profile, and will be wrapped to find their exact library dependencies,
48without impacting other applications or polluting your user environment.
49
50But Python libraries you would like to use for development cannot be installed,
51at least not individually, because they won't be able to find each other
52resulting in import errors. Instead, it is possible to create an environment
53with `python.buildEnv` or `python.withPackages` where the interpreter and other
54executables are wrapped to be able to find each other and all of the modules.
55
56In the following examples we will start by creating a simple, ad-hoc environment
57with a nix-shell that has `numpy` and `toolz` in Python 3.8; then we will create
58a re-usable environment in a single-file Python script; then we will create a
59full Python environment for development with this same environment.
60
61Philosphically, this should be familiar to users who are used to a `venv` style
62of development: individual projects create their own Python environments without
63impacting the global environment or each other.
64
65#### Ad-hoc temporary Python environment with `nix-shell` {#ad-hoc-temporary-python-environment-with-nix-shell}
66
67The simplest way to start playing with the way nix wraps and sets up Python
68environments is with `nix-shell` at the cmdline. These environments create a
69temporary shell session with a Python and a *precise* list of packages (plus
70their runtime dependencies), with no other Python packages in the Python
71interpreter's scope.
72
73To create a Python 3.8 session with `numpy` and `toolz` available, run:
74
75```sh
76$ nix-shell -p 'python38.withPackages(ps: with ps; [ numpy toolz ])'
77```
78
79By default `nix-shell` will start a `bash` session with this interpreter in our
80`PATH`, so if we then run:
81
82```Python console
83[nix-shell:~/src/nixpkgs]$ python3
84Python 3.8.1 (default, Dec 18 2019, 19:06:26)
85[GCC 9.2.0] on linux
86Type "help", "copyright", "credits" or "license" for more information.
87>>> import numpy; import toolz
88```
89
90Note that no other modules are in scope, even if they were imperatively
91installed into our user environment as a dependency of a Python application:
92
93```Python console
94>>> import requests
95Traceback (most recent call last):
96 File "<stdin>", line 1, in <module>
97ModuleNotFoundError: No module named 'requests'
98```
99
100We can add as many additional modules onto the `nix-shell` as we need, and we
101will still get 1 wrapped Python interpreter. We can start the interpreter
102directly like so:
103
104```sh
105$ nix-shell -p 'python38.withPackages(ps: with ps; [ numpy toolz requests ])' --run python3
106these derivations will be built:
107 /nix/store/xbdsrqrsfa1yva5s7pzsra8k08gxlbz1-python3-3.8.1-env.drv
108building '/nix/store/xbdsrqrsfa1yva5s7pzsra8k08gxlbz1-python3-3.8.1-env.drv'...
109created 277 symlinks in user environment
110Python 3.8.1 (default, Dec 18 2019, 19:06:26)
111[GCC 9.2.0] on linux
112Type "help", "copyright", "credits" or "license" for more information.
113>>> import requests
114>>>
115```
116
117Notice that this time it built a new Python environment, which now includes
118`requests`. Building an environment just creates wrapper scripts that expose the
119selected dependencies to the interpreter while re-using the actual modules. This
120means if any other env has installed `requests` or `numpy` in a different
121context, we don't need to recompile them -- we just recompile the wrapper script
122that sets up an interpreter pointing to them. This matters much more for "big"
123modules like `pytorch` or `tensorflow`.
124
125Module names usually match their names on [pypi.org](https://pypi.org/), but
126you can use the [Nixpkgs search website](https://nixos.org/nixos/packages.html)
127to find them as well (along with non-python packages).
128
129At this point we can create throwaway experimental Python environments with
130arbitrary dependencies. This is a good way to get a feel for how the Python
131interpreter and dependencies work in Nix and NixOS, but to do some actual
132development, we'll want to make it a bit more persistent.
133
134##### Running Python scripts and using `nix-shell` as shebang {#running-python-scripts-and-using-nix-shell-as-shebang}
135
136Sometimes, we have a script whose header looks like this:
137
138```python
139#!/usr/bin/env python3
140import numpy as np
141a = np.array([1,2])
142b = np.array([3,4])
143print(f"The dot product of {a} and {b} is: {np.dot(a, b)}")
144```
145
146Executing this script requires a `python3` that has `numpy`. Using what we learned
147in the previous section, we could startup a shell and just run it like so:
148
149```ShellSession
150$ nix-shell -p 'python38.withPackages(ps: with ps; [ numpy ])' --run 'python3 foo.py'
151The dot product of [1 2] and [3 4] is: 11
152```
153
154But if we maintain the script ourselves, and if there are more dependencies, it
155may be nice to encode those dependencies in source to make the script re-usable
156without that bit of knowledge. That can be done by using `nix-shell` as a
157[shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)), like so:
158
159```python
160#!/usr/bin/env nix-shell
161#!nix-shell -i python3 -p "python3.withPackages(ps: [ ps.numpy ])"
162import numpy as np
163a = np.array([1,2])
164b = np.array([3,4])
165print(f"The dot product of {a} and {b} is: {np.dot(a, b)}")
166```
167
168Then we simply execute it, without requiring any environment setup at all!
169
170```sh
171$ ./foo.py
172The dot product of [1 2] and [3 4] is: 11
173```
174
175If the dependencies are not available on the host where `foo.py` is executed, it
176will build or download them from a Nix binary cache prior to starting up, prior
177that it is executed on a machine with a multi-user nix installation.
178
179This provides a way to ship a self bootstrapping Python script, akin to a
180statically linked binary, where it can be run on any machine (provided nix is
181installed) without having to assume that `numpy` is installed globally on the
182system.
183
184By default it is pulling the import checkout of Nixpkgs itself from our nix
185channel, which is nice as it cache aligns with our other package builds, but we
186can make it fully reproducible by pinning the `nixpkgs` import:
187
188```python
189#!/usr/bin/env nix-shell
190#!nix-shell -i python3 -p "python3.withPackages(ps: [ ps.numpy ])"
191#!nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/d373d80b1207d52621961b16aa4a3438e4f98167.tar.gz
192import numpy as np
193a = np.array([1,2])
194b = np.array([3,4])
195print(f"The dot product of {a} and {b} is: {np.dot(a, b)}")
196```
197
198This will execute with the exact same versions of Python 3.8, numpy, and system
199dependencies a year from now as it does today, because it will always use
200exactly git commit `d373d80b1207d52621961b16aa4a3438e4f98167` of Nixpkgs for all
201of the package versions.
202
203This is also a great way to ensure the script executes identically on different
204servers.
205
206##### Load environment from `.nix` expression {#load-environment-from-.nix-expression}
207
208We've now seen how to create an ad-hoc temporary shell session, and how to
209create a single script with Python dependencies, but in the course of normal
210development we're usually working in an entire package repository.
211
212As explained in the Nix manual, `nix-shell` can also load an expression from a
213`.nix` file. Say we want to have Python 3.8, `numpy` and `toolz`, like before,
214in an environment. We can add a `shell.nix` file describing our dependencies:
215
216```nix
217with import <nixpkgs> {};
218(python38.withPackages (ps: [ps.numpy ps.toolz])).env
219```
220
221And then at the command line, just typing `nix-shell` produces the same
222environment as before. In a normal project, we'll likely have many more
223dependencies; this can provide a way for developers to share the environments
224with each other and with CI builders.
225
226What's happening here?
227
2281. We begin with importing the Nix Packages collections. `import <nixpkgs>`
229 imports the `<nixpkgs>` function, `{}` calls it and the `with` statement
230 brings all attributes of `nixpkgs` in the local scope. These attributes form
231 the main package set.
2322. Then we create a Python 3.8 environment with the `withPackages` function, as before.
2333. The `withPackages` function expects us to provide a function as an argument
234 that takes the set of all Python packages and returns a list of packages to
235 include in the environment. Here, we select the packages `numpy` and `toolz`
236 from the package set.
237
238To combine this with `mkShell` you can:
239
240```nix
241with import <nixpkgs> {};
242let
243 pythonEnv = python38.withPackages (ps: [
244 ps.numpy
245 ps.toolz
246 ]);
247in mkShell {
248 packages = [
249 pythonEnv
250
251 black
252 mypy
253
254 libffi
255 openssl
256 ];
257}
258```
259
260This will create a unified environment that has not just our Python interpreter
261and its Python dependencies, but also tools like `black` or `mypy` and libraries
262like `libffi` the `openssl` in scope. This is generic and can span any number of
263tools or languages across the Nixpkgs ecosystem.
264
265##### Installing environments globally on the system {#installing-environments-globally-on-the-system}
266
267Up to now, we've been creating environments scoped to an ad-hoc shell session,
268or a single script, or a single project. This is generally advisable, as it
269avoids pollution across contexts.
270
271However, sometimes we know we will often want a Python with some basic packages,
272and want this available without having to enter into a shell or build context.
273This can be useful to have things like vim/emacs editors and plugins or shell
274tools "just work" without having to set them up, or when running other software
275that expects packages to be installed globally.
276
277To create your own custom environment, create a file in `~/.config/nixpkgs/overlays/`
278that looks like this:
279
280```nix
281# ~/.config/nixpkgs/overlays/myEnv.nix
282self: super: {
283 myEnv = super.buildEnv {
284 name = "myEnv";
285 paths = [
286 # A Python 3 interpreter with some packages
287 (self.python3.withPackages (
288 ps: with ps; [
289 pyflakes
290 pytest
291 python-language-server
292 ]
293 ))
294
295 # Some other packages we'd like as part of this env
296 self.mypy
297 self.black
298 self.ripgrep
299 self.tmux
300 ];
301 };
302}
303```
304
305You can then build and install this to your profile with:
306
307```sh
308nix-env -iA myEnv
309```
310
311One limitation of this is that you can only have 1 Python env installed
312globally, since they conflict on the `python` to load out of your `PATH`.
313
314If you get a conflict or prefer to keep the setup clean, you can have `nix-env`
315atomically *uninstall* all other imperatively installed packages and replace
316your profile with just `myEnv` by using the `--replace` flag.
317
318##### Environment defined in `/etc/nixos/configuration.nix` {#environment-defined-in-etcnixosconfiguration.nix}
319
320For the sake of completeness, here's how to install the environment system-wide
321on NixOS.
322
323```nix
324{ # ...
325
326 environment.systemPackages = with pkgs; [
327 (python38.withPackages(ps: with ps; [ numpy toolz ]))
328 ];
329}
330```
331
332### Developing with Python {#developing-with-python}
333
334Above, we were mostly just focused on use cases and what to do to get started
335creating working Python environments in nix.
336
337Now that you know the basics to be up and running, it is time to take a step
338back and take a deeper look at how Python packages are packaged on Nix. Then,
339we will look at how you can use development mode with your code.
340
341#### Python library packages in Nixpkgs {#python-library-packages-in-nixpkgs}
342
343With Nix all packages are built by functions. The main function in Nix for
344building Python libraries is `buildPythonPackage`. Let's see how we can build the
345`toolz` package.
346
347```nix
348{ lib, buildPythonPackage, fetchPypi }:
349
350buildPythonPackage rec {
351 pname = "toolz";
352 version = "0.10.0";
353
354 src = fetchPypi {
355 inherit pname version;
356 sha256 = "08fdd5ef7c96480ad11c12d472de21acd32359996f69a5259299b540feba4560";
357 };
358
359 doCheck = false;
360
361 meta = with lib; {
362 homepage = "https://github.com/pytoolz/toolz";
363 description = "List processing tools and functional utilities";
364 license = licenses.bsd3;
365 maintainers = with maintainers; [ fridh ];
366 };
367}
368```
369
370What happens here? The function `buildPythonPackage` is called and as argument
371it accepts a set. In this case the set is a recursive set, `rec`. One of the
372arguments is the name of the package, which consists of a basename (generally
373following the name on PyPi) and a version. Another argument, `src` specifies the
374source, which in this case is fetched from PyPI using the helper function
375`fetchPypi`. The argument `doCheck` is used to set whether tests should be run
376when building the package. Furthermore, we specify some (optional) meta
377information. The output of the function is a derivation.
378
379An expression for `toolz` can be found in the Nixpkgs repository. As explained
380in the introduction of this Python section, a derivation of `toolz` is available
381for each interpreter version, e.g. `python38.pkgs.toolz` refers to the `toolz`
382derivation corresponding to the CPython 3.8 interpreter.
383
384The above example works when you're directly working on
385`pkgs/top-level/python-packages.nix` in the Nixpkgs repository. Often though,
386you will want to test a Nix expression outside of the Nixpkgs tree.
387
388The following expression creates a derivation for the `toolz` package,
389and adds it along with a `numpy` package to a Python environment.
390
391```nix
392with import <nixpkgs> {};
393
394( let
395 my_toolz = python38.pkgs.buildPythonPackage rec {
396 pname = "toolz";
397 version = "0.10.0";
398
399 src = python38.pkgs.fetchPypi {
400 inherit pname version;
401 sha256 = "08fdd5ef7c96480ad11c12d472de21acd32359996f69a5259299b540feba4560";
402 };
403
404 doCheck = false;
405
406 meta = {
407 homepage = "https://github.com/pytoolz/toolz/";
408 description = "List processing tools and functional utilities";
409 };
410 };
411
412 in python38.withPackages (ps: [ps.numpy my_toolz])
413).env
414```
415
416Executing `nix-shell` will result in an environment in which you can use
417Python 3.8 and the `toolz` package. As you can see we had to explicitly mention
418for which Python version we want to build a package.
419
420So, what did we do here? Well, we took the Nix expression that we used earlier
421to build a Python environment, and said that we wanted to include our own
422version of `toolz`, named `my_toolz`. To introduce our own package in the scope
423of `withPackages` we used a `let` expression. You can see that we used
424`ps.numpy` to select numpy from the nixpkgs package set (`ps`). We did not take
425`toolz` from the Nixpkgs package set this time, but instead took our own version
426that we introduced with the `let` expression.
427
428#### Handling dependencies {#handling-dependencies}
429
430Our example, `toolz`, does not have any dependencies on other Python packages or
431system libraries. According to the manual, `buildPythonPackage` uses the
432arguments `buildInputs` and `propagatedBuildInputs` to specify dependencies. If
433something is exclusively a build-time dependency, then the dependency should be
434included in `buildInputs`, but if it is (also) a runtime dependency, then it
435should be added to `propagatedBuildInputs`. Test dependencies are considered
436build-time dependencies and passed to `checkInputs`.
437
438The following example shows which arguments are given to `buildPythonPackage` in
439order to build [`datashape`](https://github.com/blaze/datashape).
440
441```nix
442{ lib, buildPythonPackage, fetchPypi, numpy, multipledispatch, python-dateutil, pytest }:
443
444buildPythonPackage rec {
445 pname = "datashape";
446 version = "0.4.7";
447
448 src = fetchPypi {
449 inherit pname version;
450 sha256 = "14b2ef766d4c9652ab813182e866f493475e65e558bed0822e38bf07bba1a278";
451 };
452
453 checkInputs = [ pytest ];
454 propagatedBuildInputs = [ numpy multipledispatch python-dateutil ];
455
456 meta = with lib; {
457 homepage = "https://github.com/ContinuumIO/datashape";
458 description = "A data description language";
459 license = licenses.bsd2;
460 maintainers = with maintainers; [ fridh ];
461 };
462}
463```
464
465We can see several runtime dependencies, `numpy`, `multipledispatch`, and
466`python-dateutil`. Furthermore, we have one `checkInputs`, i.e. `pytest`. `pytest` is a
467test runner and is only used during the `checkPhase` and is therefore not added
468to `propagatedBuildInputs`.
469
470In the previous case we had only dependencies on other Python packages to consider.
471Occasionally you have also system libraries to consider. E.g., `lxml` provides
472Python bindings to `libxml2` and `libxslt`. These libraries are only required
473when building the bindings and are therefore added as `buildInputs`.
474
475```nix
476{ lib, pkgs, buildPythonPackage, fetchPypi }:
477
478buildPythonPackage rec {
479 pname = "lxml";
480 version = "3.4.4";
481
482 src = fetchPypi {
483 inherit pname version;
484 sha256 = "16a0fa97hym9ysdk3rmqz32xdjqmy4w34ld3rm3jf5viqjx65lxk";
485 };
486
487 buildInputs = [ pkgs.libxml2 pkgs.libxslt ];
488
489 meta = with lib; {
490 description = "Pythonic binding for the libxml2 and libxslt libraries";
491 homepage = "https://lxml.de";
492 license = licenses.bsd3;
493 maintainers = with maintainers; [ sjourdois ];
494 };
495}
496```
497
498In this example `lxml` and Nix are able to work out exactly where the relevant
499files of the dependencies are. This is not always the case.
500
501The example below shows bindings to The Fastest Fourier Transform in the West,
502commonly known as FFTW. On Nix we have separate packages of FFTW for the
503different types of floats (`"single"`, `"double"`, `"long-double"`). The
504bindings need all three types, and therefore we add all three as `buildInputs`.
505The bindings don't expect to find each of them in a different folder, and
506therefore we have to set `LDFLAGS` and `CFLAGS`.
507
508```nix
509{ lib, pkgs, buildPythonPackage, fetchPypi, numpy, scipy }:
510
511buildPythonPackage rec {
512 pname = "pyFFTW";
513 version = "0.9.2";
514
515 src = fetchPypi {
516 inherit pname version;
517 sha256 = "f6bbb6afa93085409ab24885a1a3cdb8909f095a142f4d49e346f2bd1b789074";
518 };
519
520 buildInputs = [ pkgs.fftw pkgs.fftwFloat pkgs.fftwLongDouble];
521
522 propagatedBuildInputs = [ numpy scipy ];
523
524 # Tests cannot import pyfftw. pyfftw works fine though.
525 doCheck = false;
526
527 preConfigure = ''
528 export LDFLAGS="-L${pkgs.fftw.dev}/lib -L${pkgs.fftwFloat.out}/lib -L${pkgs.fftwLongDouble.out}/lib"
529 export CFLAGS="-I${pkgs.fftw.dev}/include -I${pkgs.fftwFloat.dev}/include -I${pkgs.fftwLongDouble.dev}/include"
530 '';
531
532 meta = with lib; {
533 description = "A pythonic wrapper around FFTW, the FFT library, presenting a unified interface for all the supported transforms";
534 homepage = "http://hgomersall.github.com/pyFFTW";
535 license = with licenses; [ bsd2 bsd3 ];
536 maintainers = with maintainers; [ fridh ];
537 };
538}
539```
540
541Note also the line `doCheck = false;`, we explicitly disabled running the test-suite.
542
543#### Testing Python Packages {#testing-python-packages}
544
545It is highly encouraged to have testing as part of the package build. This
546helps to avoid situations where the package was able to build and install,
547but is not usable at runtime. Currently, all packages will use the `test`
548command provided by the setup.py (i.e. `python setup.py test`). However,
549this is currently deprecated https://github.com/pypa/setuptools/pull/1878
550and your package should provide its own checkPhase.
551
552*NOTE:* The `checkPhase` for python maps to the `installCheckPhase` on a
553normal derivation. This is due to many python packages not behaving well
554to the pre-installed version of the package. Version info, and natively
555compiled extensions generally only exist in the install directory, and
556thus can cause issues when a test suite asserts on that behavior.
557
558*NOTE:* Tests should only be disabled if they don't agree with nix
559(e.g. external dependencies, network access, flakey tests), however,
560as many tests should be enabled as possible. Failing tests can still be
561a good indication that the package is not in a valid state.
562
563#### Using pytest {#using-pytest}
564
565Pytest is the most common test runner for python repositories. A trivial
566test run would be:
567
568```
569 checkInputs = [ pytest ];
570 checkPhase = "pytest";
571```
572
573However, many repositories' test suites do not translate well to nix's build
574sandbox, and will generally need many tests to be disabled.
575
576To filter tests using pytest, one can do the following:
577
578```
579 checkInputs = [ pytest ];
580 # avoid tests which need additional data or touch network
581 checkPhase = ''
582 pytest tests/ --ignore=tests/integration -k 'not download and not update'
583 '';
584```
585
586`--ignore` will tell pytest to ignore that file or directory from being
587collected as part of a test run. This is useful is a file uses a package
588which is not available in nixpkgs, thus skipping that test file is much
589easier than having to create a new package.
590
591`-k` is used to define a predicate for test names. In this example, we are
592filtering out tests which contain `download` or `update` in their test case name.
593Only one `-k` argument is allowed, and thus a long predicate should be concatenated
594with “\\” and wrapped to the next line.
595
596*NOTE:* In pytest==6.0.1, the use of “\\” to continue a line (e.g. `-k 'not download \'`) has
597been removed, in this case, it's recommended to use `pytestCheckHook`.
598
599#### Using pytestCheckHook {#using-pytestcheckhook}
600
601`pytestCheckHook` is a convenient hook which will substitute the setuptools
602`test` command for a checkPhase which runs `pytest`. This is also beneficial
603when a package may need many items disabled to run the test suite.
604
605Using the example above, the analagous pytestCheckHook usage would be:
606
607```
608 checkInputs = [ pytestCheckHook ];
609
610 # requires additional data
611 pytestFlagsArray = [ "tests/" "--ignore=tests/integration" ];
612
613 disabledTests = [
614 # touches network
615 "download"
616 "update"
617 ];
618
619 disabledTestPaths = [
620 "tests/test_failing.py"
621 ];
622```
623
624This is expecially useful when tests need to be conditionallydisabled,
625for example:
626
627```
628 disabledTests = [
629 # touches network
630 "download"
631 "update"
632 ] ++ lib.optionals (pythonAtLeast "3.8") [
633 # broken due to python3.8 async changes
634 "async"
635 ] ++ lib.optionals stdenv.isDarwin [
636 # can fail when building with other packages
637 "socket"
638 ];
639```
640Trying to concatenate the related strings to disable tests in a regular checkPhase
641would be much harder to read. This also enables us to comment on why specific tests
642are disabled.
643
644#### Using pythonImportsCheck {#using-pythonimportscheck}
645
646Although unit tests are highly prefered to validate correctness of a package, not
647all packages have test suites that can be ran easily, and some have none at all.
648To help ensure the package still works, `pythonImportsCheck` can attempt to import
649the listed modules.
650
651```
652 pythonImportsCheck = [ "requests" "urllib" ];
653```
654roughly translates to:
655```
656 postCheck = ''
657 PYTHONPATH=$out/${python.sitePackages}:$PYTHONPATH
658 python -c "import requests; import urllib"
659 '';
660```
661However, this is done in it's own phase, and not dependent on whether `doCheck = true;`
662
663This can also be useful in verifying that the package doesn't assume commonly
664present packages (e.g. `setuptools`)
665
666### Develop local package {#develop-local-package}
667
668As a Python developer you're likely aware of [development mode](http://setuptools.readthedocs.io/en/latest/setuptools.html#development-mode)
669(`python setup.py develop`); instead of installing the package this command
670creates a special link to the project code. That way, you can run updated code
671without having to reinstall after each and every change you make. Development
672mode is also available. Let's see how you can use it.
673
674In the previous Nix expression the source was fetched from an url. We can also
675refer to a local source instead using `src = ./path/to/source/tree;`
676
677If we create a `shell.nix` file which calls `buildPythonPackage`, and if `src`
678is a local source, and if the local source has a `setup.py`, then development
679mode is activated.
680
681In the following example we create a simple environment that has a Python 3.8
682version of our package in it, as well as its dependencies and other packages we
683like to have in the environment, all specified with `propagatedBuildInputs`.
684Indeed, we can just add any package we like to have in our environment to
685`propagatedBuildInputs`.
686
687```nix
688with import <nixpkgs> {};
689with python38Packages;
690
691buildPythonPackage rec {
692 name = "mypackage";
693 src = ./path/to/package/source;
694 propagatedBuildInputs = [ pytest numpy pkgs.libsndfile ];
695}
696```
697
698It is important to note that due to how development mode is implemented on Nix
699it is not possible to have multiple packages simultaneously in development mode.
700
701### Organising your packages {#organising-your-packages}
702
703So far we discussed how you can use Python on Nix, and how you can develop with
704it. We've looked at how you write expressions to package Python packages, and we
705looked at how you can create environments in which specified packages are
706available.
707
708At some point you'll likely have multiple packages which you would
709like to be able to use in different projects. In order to minimise unnecessary
710duplication we now look at how you can maintain a repository with your
711own packages. The important functions here are `import` and `callPackage`.
712
713### Including a derivation using `callPackage` {#including-a-derivation-using-callpackage}
714
715Earlier we created a Python environment using `withPackages`, and included the
716`toolz` package via a `let` expression.
717Let's split the package definition from the environment definition.
718
719We first create a function that builds `toolz` in `~/path/to/toolz/release.nix`
720
721```nix
722{ lib, buildPythonPackage }:
723
724buildPythonPackage rec {
725 pname = "toolz";
726 version = "0.10.0";
727
728 src = fetchPypi {
729 inherit pname version;
730 sha256 = "08fdd5ef7c96480ad11c12d472de21acd32359996f69a5259299b540feba4560";
731 };
732
733 meta = with lib; {
734 homepage = "https://github.com/pytoolz/toolz/";
735 description = "List processing tools and functional utilities";
736 license = licenses.bsd3;
737 maintainers = with maintainers; [ fridh ];
738 };
739}
740```
741
742It takes an argument `buildPythonPackage`. We now call this function using
743`callPackage` in the definition of our environment
744
745```nix
746with import <nixpkgs> {};
747
748( let
749 toolz = callPackage /path/to/toolz/release.nix {
750 buildPythonPackage = python38Packages.buildPythonPackage;
751 };
752 in python38.withPackages (ps: [ ps.numpy toolz ])
753).env
754```
755
756Important to remember is that the Python version for which the package is made
757depends on the `python` derivation that is passed to `buildPythonPackage`. Nix
758tries to automatically pass arguments when possible, which is why generally you
759don't explicitly define which `python` derivation should be used. In the above
760example we use `buildPythonPackage` that is part of the set `python38Packages`,
761and in this case the `python38` interpreter is automatically used.
762
763## Reference {#reference}
764
765### Interpreters {#interpreters}
766
767Versions 2.7, 3.6, 3.7, 3.8 and 3.9 of the CPython interpreter are available as
768respectively `python27`, `python37`, `python38` and `python39`. The
769aliases `python2` and `python3` correspond to respectively `python27` and
770`python39`. The attribute `python` maps to `python2`. The PyPy interpreters
771compatible with Python 2.7 and 3 are available as `pypy27` and `pypy3`, with
772aliases `pypy2` mapping to `pypy27` and `pypy` mapping to `pypy2`. The Nix
773expressions for the interpreters can be found in
774`pkgs/development/interpreters/python`.
775
776All packages depending on any Python interpreter get appended
777`out/{python.sitePackages}` to `$PYTHONPATH` if such directory
778exists.
779
780#### Missing `tkinter` module standard library {#missing-tkinter-module-standard-library}
781
782To reduce closure size the `Tkinter`/`tkinter` is available as a separate package, `pythonPackages.tkinter`.
783
784#### Attributes on interpreters packages {#attributes-on-interpreters-packages}
785
786Each interpreter has the following attributes:
787
788- `libPrefix`. Name of the folder in `${python}/lib/` for corresponding interpreter.
789- `interpreter`. Alias for `${python}/bin/${executable}`.
790- `buildEnv`. Function to build python interpreter environments with extra packages bundled together. See section *python.buildEnv function* for usage and documentation.
791- `withPackages`. Simpler interface to `buildEnv`. See section *python.withPackages function* for usage and documentation.
792- `sitePackages`. Alias for `lib/${libPrefix}/site-packages`.
793- `executable`. Name of the interpreter executable, e.g. `python3.8`.
794- `pkgs`. Set of Python packages for that specific interpreter. The package set can be modified by overriding the interpreter and passing `packageOverrides`.
795
796### Optimizations {#optimizations}
797
798The Python interpreters are by default not build with optimizations enabled, because
799the builds are in that case not reproducible. To enable optimizations, override the
800interpreter of interest, e.g using
801
802```
803let
804 pkgs = import ./. {};
805 mypython = pkgs.python3.override {
806 enableOptimizations = true;
807 reproducibleBuild = false;
808 self = mypython;
809 };
810in mypython
811```
812
813### Building packages and applications {#building-packages-and-applications}
814
815Python libraries and applications that use `setuptools` or
816`distutils` are typically built with respectively the `buildPythonPackage` and
817`buildPythonApplication` functions. These two functions also support installing a `wheel`.
818
819All Python packages reside in `pkgs/top-level/python-packages.nix` and all
820applications elsewhere. In case a package is used as both a library and an
821application, then the package should be in `pkgs/top-level/python-packages.nix`
822since only those packages are made available for all interpreter versions. The
823preferred location for library expressions is in
824`pkgs/development/python-modules`. It is important that these packages are
825called from `pkgs/top-level/python-packages.nix` and not elsewhere, to guarantee
826the right version of the package is built.
827
828Based on the packages defined in `pkgs/top-level/python-packages.nix` an
829attribute set is created for each available Python interpreter. The available
830sets are
831
832* `pkgs.python27Packages`
833* `pkgs.python37Packages`
834* `pkgs.python38Packages`
835* `pkgs.python39Packages`
836* `pkgs.python310Packages`
837* `pkgs.pypyPackages`
838
839and the aliases
840
841* `pkgs.python2Packages` pointing to `pkgs.python27Packages`
842* `pkgs.python3Packages` pointing to `pkgs.python39Packages`
843* `pkgs.pythonPackages` pointing to `pkgs.python2Packages`
844
845#### `buildPythonPackage` function {#buildpythonpackage-function}
846
847The `buildPythonPackage` function is implemented in
848`pkgs/development/interpreters/python/mk-python-derivation`
849using setup hooks.
850
851The following is an example:
852
853```nix
854{ lib, buildPythonPackage, fetchPypi, hypothesis, setuptools-scm, attrs, py, setuptools, six, pluggy }:
855
856buildPythonPackage rec {
857 pname = "pytest";
858 version = "3.3.1";
859
860 src = fetchPypi {
861 inherit pname version;
862 sha256 = "cf8436dc59d8695346fcd3ab296de46425ecab00d64096cebe79fb51ecb2eb93";
863 };
864
865 postPatch = ''
866 # don't test bash builtins
867 rm testing/test_argcomplete.py
868 '';
869
870 checkInputs = [ hypothesis ];
871 nativeBuildInputs = [ setuptools-scm ];
872 propagatedBuildInputs = [ attrs py setuptools six pluggy ];
873
874 meta = with lib; {
875 maintainers = with maintainers; [ domenkozar lovek323 madjar lsix ];
876 description = "Framework for writing tests";
877 };
878}
879```
880
881The `buildPythonPackage` mainly does four things:
882
883* In the `buildPhase`, it calls `${python.interpreter} setup.py bdist_wheel` to
884 build a wheel binary zipfile.
885* In the `installPhase`, it installs the wheel file using `pip install *.whl`.
886* In the `postFixup` phase, the `wrapPythonPrograms` bash function is called to
887 wrap all programs in the `$out/bin/*` directory to include `$PATH`
888 environment variable and add dependent libraries to script's `sys.path`.
889* In the `installCheck` phase, `${python.interpreter} setup.py test` is ran.
890
891By default tests are run because `doCheck = true`. Test dependencies, like
892e.g. the test runner, should be added to `checkInputs`.
893
894By default `meta.platforms` is set to the same value
895as the interpreter unless overridden otherwise.
896
897##### `buildPythonPackage` parameters {#buildpythonpackage-parameters}
898
899All parameters from `stdenv.mkDerivation` function are still supported. The
900following are specific to `buildPythonPackage`:
901
902* `catchConflicts ? true`: If `true`, abort package build if a package name
903 appears more than once in dependency tree. Default is `true`.
904* `disabled` ? false: If `true`, package is not built for the particular Python
905 interpreter version.
906* `dontWrapPythonPrograms ? false`: Skip wrapping of Python programs.
907* `permitUserSite ? false`: Skip setting the `PYTHONNOUSERSITE` environment
908 variable in wrapped programs.
909* `format ? "setuptools"`: Format of the source. Valid options are
910 `"setuptools"`, `"pyproject"`, `"flit"`, `"wheel"`, and `"other"`.
911 `"setuptools"` is for when the source has a `setup.py` and `setuptools` is
912 used to build a wheel, `flit`, in case `flit` should be used to build a wheel,
913 and `wheel` in case a wheel is provided. Use `other` when a custom
914 `buildPhase` and/or `installPhase` is needed.
915* `makeWrapperArgs ? []`: A list of strings. Arguments to be passed to
916 `makeWrapper`, which wraps generated binaries. By default, the arguments to
917 `makeWrapper` set `PATH` and `PYTHONPATH` environment variables before calling
918 the binary. Additional arguments here can allow a developer to set environment
919 variables which will be available when the binary is run. For example,
920 `makeWrapperArgs = ["--set FOO BAR" "--set BAZ QUX"]`.
921* `namePrefix`: Prepends text to `${name}` parameter. In case of libraries, this
922 defaults to `"python3.8-"` for Python 3.8, etc., and in case of applications
923 to `""`.
924* `pipInstallFlags ? []`: A list of strings. Arguments to be passed to `pip
925 install`. To pass options to `python setup.py install`, use
926 `--install-option`. E.g., `pipInstallFlags=["--install-option='--cpp_implementation'"]`.
927* `pythonPath ? []`: List of packages to be added into `$PYTHONPATH`. Packages
928 in `pythonPath` are not propagated (contrary to `propagatedBuildInputs`).
929* `preShellHook`: Hook to execute commands before `shellHook`.
930* `postShellHook`: Hook to execute commands after `shellHook`.
931* `removeBinByteCode ? true`: Remove bytecode from `/bin`. Bytecode is only
932 created when the filenames end with `.py`.
933* `setupPyGlobalFlags ? []`: List of flags passed to `setup.py` command.
934* `setupPyBuildFlags ? []`: List of flags passed to `setup.py build_ext` command.
935
936The `stdenv.mkDerivation` function accepts various parameters for describing
937build inputs (see "Specifying dependencies"). The following are of special
938interest for Python packages, either because these are primarily used, or
939because their behaviour is different:
940
941* `nativeBuildInputs ? []`: Build-time only dependencies. Typically executables
942 as well as the items listed in `setup_requires`.
943* `buildInputs ? []`: Build and/or run-time dependencies that need to be
944 compiled for the host machine. Typically non-Python libraries which are being
945 linked.
946* `checkInputs ? []`: Dependencies needed for running the `checkPhase`. These
947 are added to `nativeBuildInputs` when `doCheck = true`. Items listed in
948 `tests_require` go here.
949* `propagatedBuildInputs ? []`: Aside from propagating dependencies,
950 `buildPythonPackage` also injects code into and wraps executables with the
951 paths included in this list. Items listed in `install_requires` go here.
952
953##### Overriding Python packages {#overriding-python-packages}
954
955The `buildPythonPackage` function has a `overridePythonAttrs` method that can be
956used to override the package. In the following example we create an environment
957where we have the `blaze` package using an older version of `pandas`. We
958override first the Python interpreter and pass `packageOverrides` which contains
959the overrides for packages in the package set.
960
961```nix
962with import <nixpkgs> {};
963
964(let
965 python = let
966 packageOverrides = self: super: {
967 pandas = super.pandas.overridePythonAttrs(old: rec {
968 version = "0.19.1";
969 src = super.fetchPypi {
970 pname = "pandas";
971 inherit version;
972 sha256 = "08blshqj9zj1wyjhhw3kl2vas75vhhicvv72flvf1z3jvapgw295";
973 };
974 });
975 };
976 in pkgs.python3.override {inherit packageOverrides; self = python;};
977
978in python.withPackages(ps: [ps.blaze])).env
979```
980
981#### `buildPythonApplication` function {#buildpythonapplication-function}
982
983The `buildPythonApplication` function is practically the same as
984`buildPythonPackage`. The main purpose of this function is to build a Python
985package where one is interested only in the executables, and not importable
986modules. For that reason, when adding this package to a `python.buildEnv`, the
987modules won't be made available.
988
989Another difference is that `buildPythonPackage` by default prefixes the names of
990the packages with the version of the interpreter. Because this is irrelevant for
991applications, the prefix is omitted.
992
993When packaging a Python application with `buildPythonApplication`, it should be
994called with `callPackage` and passed `python` or `pythonPackages` (possibly
995specifying an interpreter version), like this:
996
997```nix
998{ lib, python3Packages }:
999
1000python3Packages.buildPythonApplication rec {
1001 pname = "luigi";
1002 version = "2.7.9";
1003
1004 src = python3Packages.fetchPypi {
1005 inherit pname version;
1006 sha256 = "035w8gqql36zlan0xjrzz9j4lh9hs0qrsgnbyw07qs7lnkvbdv9x";
1007 };
1008
1009 propagatedBuildInputs = with python3Packages; [ tornado_4 python-daemon ];
1010
1011 meta = with lib; {
1012 ...
1013 };
1014}
1015```
1016
1017This is then added to `all-packages.nix` just as any other application would be.
1018
1019```nix
1020luigi = callPackage ../applications/networking/cluster/luigi { };
1021```
1022
1023Since the package is an application, a consumer doesn't need to care about
1024Python versions or modules, which is why they don't go in `pythonPackages`.
1025
1026#### `toPythonApplication` function {#topythonapplication-function}
1027
1028A distinction is made between applications and libraries, however, sometimes a
1029package is used as both. In this case the package is added as a library to
1030`python-packages.nix` and as an application to `all-packages.nix`. To reduce
1031duplication the `toPythonApplication` can be used to convert a library to an
1032application.
1033
1034The Nix expression shall use `buildPythonPackage` and be called from
1035`python-packages.nix`. A reference shall be created from `all-packages.nix` to
1036the attribute in `python-packages.nix`, and the `toPythonApplication` shall be
1037applied to the reference:
1038
1039```nix
1040youtube-dl = with pythonPackages; toPythonApplication youtube-dl;
1041```
1042
1043#### `toPythonModule` function {#topythonmodule-function}
1044
1045In some cases, such as bindings, a package is created using
1046`stdenv.mkDerivation` and added as attribute in `all-packages.nix`. The Python
1047bindings should be made available from `python-packages.nix`. The
1048`toPythonModule` function takes a derivation and makes certain Python-specific
1049modifications.
1050
1051```nix
1052opencv = toPythonModule (pkgs.opencv.override {
1053 enablePython = true;
1054 pythonPackages = self;
1055});
1056```
1057
1058Do pay attention to passing in the right Python version!
1059
1060#### `python.buildEnv` function {#python.buildenv-function}
1061
1062Python environments can be created using the low-level `pkgs.buildEnv` function.
1063This example shows how to create an environment that has the Pyramid Web Framework.
1064Saving the following as `default.nix`
1065
1066```nix
1067with import <nixpkgs> {};
1068
1069python.buildEnv.override {
1070 extraLibs = [ pythonPackages.pyramid ];
1071 ignoreCollisions = true;
1072}
1073```
1074
1075and running `nix-build` will create
1076
1077```
1078/nix/store/cf1xhjwzmdki7fasgr4kz6di72ykicl5-python-2.7.8-env
1079```
1080
1081with wrapped binaries in `bin/`.
1082
1083You can also use the `env` attribute to create local environments with needed
1084packages installed. This is somewhat comparable to `virtualenv`. For example,
1085running `nix-shell` with the following `shell.nix`
1086
1087```nix
1088with import <nixpkgs> {};
1089
1090(python3.buildEnv.override {
1091 extraLibs = with python3Packages; [ numpy requests ];
1092}).env
1093```
1094
1095will drop you into a shell where Python will have the
1096specified packages in its path.
1097
1098##### `python.buildEnv` arguments {#python.buildenv-arguments}
1099
1100
1101* `extraLibs`: List of packages installed inside the environment.
1102* `postBuild`: Shell command executed after the build of environment.
1103* `ignoreCollisions`: Ignore file collisions inside the environment (default is `false`).
1104* `permitUserSite`: Skip setting the `PYTHONNOUSERSITE` environment variable in
1105 wrapped binaries in the environment.
1106
1107#### `python.withPackages` function {#python.withpackages-function}
1108
1109The `python.withPackages` function provides a simpler interface to the `python.buildEnv` functionality.
1110It takes a function as an argument that is passed the set of python packages and returns the list
1111of the packages to be included in the environment. Using the `withPackages` function, the previous
1112example for the Pyramid Web Framework environment can be written like this:
1113
1114```nix
1115with import <nixpkgs> {};
1116
1117python.withPackages (ps: [ps.pyramid])
1118```
1119
1120`withPackages` passes the correct package set for the specific interpreter
1121version as an argument to the function. In the above example, `ps` equals
1122`pythonPackages`. But you can also easily switch to using python3:
1123
1124```nix
1125with import <nixpkgs> {};
1126
1127python3.withPackages (ps: [ps.pyramid])
1128```
1129
1130Now, `ps` is set to `python3Packages`, matching the version of the interpreter.
1131
1132As `python.withPackages` simply uses `python.buildEnv` under the hood, it also
1133supports the `env` attribute. The `shell.nix` file from the previous section can
1134thus be also written like this:
1135
1136```nix
1137with import <nixpkgs> {};
1138
1139(python38.withPackages (ps: [ps.numpy ps.requests])).env
1140```
1141
1142In contrast to `python.buildEnv`, `python.withPackages` does not support the
1143more advanced options such as `ignoreCollisions = true` or `postBuild`. If you
1144need them, you have to use `python.buildEnv`.
1145
1146Python 2 namespace packages may provide `__init__.py` that collide. In that case
1147`python.buildEnv` should be used with `ignoreCollisions = true`.
1148
1149#### Setup hooks {#setup-hooks}
1150
1151The following are setup hooks specifically for Python packages. Most of these
1152are used in `buildPythonPackage`.
1153
1154- `eggUnpackhook` to move an egg to the correct folder so it can be installed
1155 with the `eggInstallHook`
1156- `eggBuildHook` to skip building for eggs.
1157- `eggInstallHook` to install eggs.
1158- `flitBuildHook` to build a wheel using `flit`.
1159- `pipBuildHook` to build a wheel using `pip` and PEP 517. Note a build system
1160 (e.g. `setuptools` or `flit`) should still be added as `nativeBuildInput`.
1161- `pipInstallHook` to install wheels.
1162- `pytestCheckHook` to run tests with `pytest`. See [example usage](#using-pytestcheckhook).
1163- `pythonCatchConflictsHook` to check whether a Python package is not already existing.
1164- `pythonImportsCheckHook` to check whether importing the listed modules works.
1165- `pythonRemoveBinBytecode` to remove bytecode from the `/bin` folder.
1166- `setuptoolsBuildHook` to build a wheel using `setuptools`.
1167- `setuptoolsCheckHook` to run tests with `python setup.py test`.
1168- `venvShellHook` to source a Python 3 `venv` at the `venvDir` location. A
1169 `venv` is created if it does not yet exist. `postVenvCreation` can be used to
1170 to run commands only after venv is first created.
1171- `wheelUnpackHook` to move a wheel to the correct folder so it can be installed
1172 with the `pipInstallHook`.
1173
1174### Development mode {#development-mode}
1175
1176Development or editable mode is supported. To develop Python packages
1177`buildPythonPackage` has additional logic inside `shellPhase` to run `pip
1178install -e . --prefix $TMPDIR/`for the package.
1179
1180Warning: `shellPhase` is executed only if `setup.py` exists.
1181
1182Given a `default.nix`:
1183
1184```nix
1185with import <nixpkgs> {};
1186
1187pythonPackages.buildPythonPackage {
1188 name = "myproject";
1189 buildInputs = with pythonPackages; [ pyramid ];
1190
1191 src = ./.;
1192}
1193```
1194
1195Running `nix-shell` with no arguments should give you the environment in which
1196the package would be built with `nix-build`.
1197
1198Shortcut to setup environments with C headers/libraries and Python packages:
1199
1200```shell
1201nix-shell -p pythonPackages.pyramid zlib libjpeg git
1202```
1203
1204Note: There is a boolean value `lib.inNixShell` set to `true` if nix-shell is invoked.
1205
1206### Tools {#tools}
1207
1208Packages inside nixpkgs are written by hand. However many tools exist in
1209community to help save time. No tool is preferred at the moment.
1210
1211- [pypi2nix](https://github.com/nix-community/pypi2nix): Generate Nix
1212 expressions for your Python project. Note that [sharing derivations from
1213 pypi2nix with nixpkgs is possible but not
1214 encouraged](https://github.com/nix-community/pypi2nix/issues/222#issuecomment-443497376).
1215- [nixpkgs-pytools](https://github.com/nix-community/nixpkgs-pytools)
1216- [poetry2nix](https://github.com/nix-community/poetry2nix)
1217
1218### Deterministic builds {#deterministic-builds}
1219
1220The Python interpreters are now built deterministically. Minor modifications had
1221to be made to the interpreters in order to generate deterministic bytecode. This
1222has security implications and is relevant for those using Python in a
1223`nix-shell`.
1224
1225When the environment variable `DETERMINISTIC_BUILD` is set, all bytecode will
1226have timestamp 1. The `buildPythonPackage` function sets `DETERMINISTIC_BUILD=1`
1227and [PYTHONHASHSEED=0](https://docs.python.org/3.8/using/cmdline.html#envvar-PYTHONHASHSEED).
1228Both are also exported in `nix-shell`.
1229
1230### Automatic tests {#automatic-tests}
1231
1232It is recommended to test packages as part of the build process.
1233Source distributions (`sdist`) often include test files, but not always.
1234
1235By default the command `python setup.py test` is run as part of the
1236`checkPhase`, but often it is necessary to pass a custom `checkPhase`. An
1237example of such a situation is when `py.test` is used.
1238
1239#### Common issues {#common-issues}
1240
1241* Non-working tests can often be deselected. By default `buildPythonPackage`
1242 runs `python setup.py test`. Most Python modules follows the standard test
1243 protocol where the pytest runner can be used instead. `py.test` supports a
1244 `-k` parameter to ignore test methods or classes:
1245
1246 ```nix
1247 buildPythonPackage {
1248 # ...
1249 # assumes the tests are located in tests
1250 checkInputs = [ pytest ];
1251 checkPhase = ''
1252 py.test -k 'not function_name and not other_function' tests
1253 '';
1254 }
1255 ```
1256
1257* Tests that attempt to access `$HOME` can be fixed by using the following
1258 work-around before running tests (e.g. `preCheck`): `export HOME=$(mktemp -d)`
1259
1260## FAQ {#faq}
1261
1262### How to solve circular dependencies? {#how-to-solve-circular-dependencies}
1263
1264Consider the packages `A` and `B` that depend on each other. When packaging `B`,
1265a solution is to override package `A` not to depend on `B` as an input. The same
1266should also be done when packaging `A`.
1267
1268### How to override a Python package? {#how-to-override-a-python-package}
1269
1270We can override the interpreter and pass `packageOverrides`. In the following
1271example we rename the `pandas` package and build it.
1272
1273```nix
1274with import <nixpkgs> {};
1275
1276(let
1277 python = let
1278 packageOverrides = self: super: {
1279 pandas = super.pandas.overridePythonAttrs(old: {name="foo";});
1280 };
1281 in pkgs.python38.override {inherit packageOverrides;};
1282
1283in python.withPackages(ps: [ps.pandas])).env
1284```
1285
1286Using `nix-build` on this expression will build an environment that contains the
1287package `pandas` but with the new name `foo`.
1288
1289All packages in the package set will use the renamed package. A typical use case
1290is to switch to another version of a certain package. For example, in the
1291Nixpkgs repository we have multiple versions of `django` and `scipy`. In the
1292following example we use a different version of `scipy` and create an
1293environment that uses it. All packages in the Python package set will now use
1294the updated `scipy` version.
1295
1296```nix
1297with import <nixpkgs> {};
1298
1299( let
1300 packageOverrides = self: super: {
1301 scipy = super.scipy_0_17;
1302 };
1303 in (pkgs.python38.override {inherit packageOverrides;}).withPackages (ps: [ps.blaze])
1304).env
1305```
1306
1307The requested package `blaze` depends on `pandas` which itself depends on `scipy`.
1308
1309If you want the whole of Nixpkgs to use your modifications, then you can use
1310`overlays` as explained in this manual. In the following example we build a
1311`inkscape` using a different version of `numpy`.
1312
1313```nix
1314let
1315 pkgs = import <nixpkgs> {};
1316 newpkgs = import pkgs.path { overlays = [ (self: super: {
1317 python38 = let
1318 packageOverrides = python-self: python-super: {
1319 numpy = python-super.numpy_1_18;
1320 };
1321 in super.python38.override {inherit packageOverrides;};
1322 } ) ]; };
1323in newpkgs.inkscape
1324```
1325
1326### `python setup.py bdist_wheel` cannot create .whl {#python-setup.py-bdist_wheel-cannot-create-.whl}
1327
1328Executing `python setup.py bdist_wheel` in a `nix-shell `fails with
1329```
1330ValueError: ZIP does not support timestamps before 1980
1331```
1332
1333This is because files from the Nix store (which have a timestamp of the UNIX
1334epoch of January 1, 1970) are included in the .ZIP, but .ZIP archives follow the
1335DOS convention of counting timestamps from 1980.
1336
1337The command `bdist_wheel` reads the `SOURCE_DATE_EPOCH` environment variable,
1338which `nix-shell` sets to 1. Unsetting this variable or giving it a value
1339corresponding to 1980 or later enables building wheels.
1340
1341Use 1980 as timestamp:
1342
1343```shell
1344nix-shell --run "SOURCE_DATE_EPOCH=315532800 python3 setup.py bdist_wheel"
1345```
1346
1347or the current time:
1348
1349```shell
1350nix-shell --run "SOURCE_DATE_EPOCH=$(date +%s) python3 setup.py bdist_wheel"
1351```
1352
1353or unset `SOURCE_DATE_EPOCH`:
1354
1355```shell
1356nix-shell --run "unset SOURCE_DATE_EPOCH; python3 setup.py bdist_wheel"
1357```
1358
1359### `install_data` / `data_files` problems {#install_data-data_files-problems}
1360
1361If you get the following error:
1362
1363```
1364could not create '/nix/store/6l1bvljpy8gazlsw2aw9skwwp4pmvyxw-python-2.7.8/etc':
1365Permission denied
1366```
1367
1368This is a [known bug](https://github.com/pypa/setuptools/issues/130) in
1369`setuptools`. Setuptools `install_data` does not respect `--prefix`. An example
1370of such package using the feature is `pkgs/tools/X11/xpra/default.nix`.
1371
1372As workaround install it as an extra `preInstall` step:
1373
1374```shell
1375${python.interpreter} setup.py install_data --install-dir=$out --root=$out
1376sed -i '/ = data\_files/d' setup.py
1377```
1378
1379### Rationale of non-existent global site-packages {#rationale-of-non-existent-global-site-packages}
1380
1381On most operating systems a global `site-packages` is maintained. This however
1382becomes problematic if you want to run multiple Python versions or have multiple
1383versions of certain libraries for your projects. Generally, you would solve such
1384issues by creating virtual environments using `virtualenv`.
1385
1386On Nix each package has an isolated dependency tree which, in the case of
1387Python, guarantees the right versions of the interpreter and libraries or
1388packages are available. There is therefore no need to maintain a global `site-packages`.
1389
1390If you want to create a Python environment for development, then the recommended
1391method is to use `nix-shell`, either with or without the `python.buildEnv`
1392function.
1393
1394### How to consume Python modules using pip in a virtual environment like I am used to on other Operating Systems? {#how-to-consume-python-modules-using-pip-in-a-virtual-environment-like-i-am-used-to-on-other-operating-systems}
1395
1396While this approach is not very idiomatic from Nix perspective, it can still be
1397useful when dealing with pre-existing projects or in situations where it's not
1398feasible or desired to write derivations for all required dependencies.
1399
1400This is an example of a `default.nix` for a `nix-shell`, which allows to consume
1401a virtual environment created by `venv`, and install Python modules through
1402`pip` the traditional way.
1403
1404Create this `default.nix` file, together with a `requirements.txt` and simply
1405execute `nix-shell`.
1406
1407```nix
1408with import <nixpkgs> { };
1409
1410let
1411 pythonPackages = python3Packages;
1412in pkgs.mkShell rec {
1413 name = "impurePythonEnv";
1414 venvDir = "./.venv";
1415 buildInputs = [
1416 # A Python interpreter including the 'venv' module is required to bootstrap
1417 # the environment.
1418 pythonPackages.python
1419
1420 # This execute some shell code to initialize a venv in $venvDir before
1421 # dropping into the shell
1422 pythonPackages.venvShellHook
1423
1424 # Those are dependencies that we would like to use from nixpkgs, which will
1425 # add them to PYTHONPATH and thus make them accessible from within the venv.
1426 pythonPackages.numpy
1427 pythonPackages.requests
1428
1429 # In this particular example, in order to compile any binary extensions they may
1430 # require, the Python modules listed in the hypothetical requirements.txt need
1431 # the following packages to be installed locally:
1432 taglib
1433 openssl
1434 git
1435 libxml2
1436 libxslt
1437 libzip
1438 zlib
1439 ];
1440
1441 # Run this command, only after creating the virtual environment
1442 postVenvCreation = ''
1443 unset SOURCE_DATE_EPOCH
1444 pip install -r requirements.txt
1445 '';
1446
1447 # Now we can execute any commands within the virtual environment.
1448 # This is optional and can be left out to run pip manually.
1449 postShellHook = ''
1450 # allow pip to install wheels
1451 unset SOURCE_DATE_EPOCH
1452 '';
1453
1454}
1455```
1456
1457In case the supplied venvShellHook is insufficient, or when Python 2 support is
1458needed, you can define your own shell hook and adapt to your needs like in the
1459following example:
1460
1461```nix
1462with import <nixpkgs> { };
1463
1464let
1465 venvDir = "./.venv";
1466 pythonPackages = python3Packages;
1467in pkgs.mkShell rec {
1468 name = "impurePythonEnv";
1469 buildInputs = [
1470 pythonPackages.python
1471 # Needed when using python 2.7
1472 # pythonPackages.virtualenv
1473 # ...
1474 ];
1475
1476 # This is very close to how venvShellHook is implemented, but
1477 # adapted to use 'virtualenv'
1478 shellHook = ''
1479 SOURCE_DATE_EPOCH=$(date +%s)
1480
1481 if [ -d "${venvDir}" ]; then
1482 echo "Skipping venv creation, '${venvDir}' already exists"
1483 else
1484 echo "Creating new venv environment in path: '${venvDir}'"
1485 # Note that the module venv was only introduced in python 3, so for 2.7
1486 # this needs to be replaced with a call to virtualenv
1487 ${pythonPackages.python.interpreter} -m venv "${venvDir}"
1488 fi
1489
1490 # Under some circumstances it might be necessary to add your virtual
1491 # environment to PYTHONPATH, which you can do here too;
1492 # PYTHONPATH=$PWD/${venvDir}/${pythonPackages.python.sitePackages}/:$PYTHONPATH
1493
1494 source "${venvDir}/bin/activate"
1495
1496 # As in the previous example, this is optional.
1497 pip install -r requirements.txt
1498 '';
1499}
1500```
1501
1502Note that the `pip install` is an imperative action. So every time `nix-shell`
1503is executed it will attempt to download the Python modules listed in
1504requirements.txt. However these will be cached locally within the `virtualenv`
1505folder and not downloaded again.
1506
1507### How to override a Python package from `configuration.nix`? {#how-to-override-a-python-package-from-configuration.nix}
1508
1509If you need to change a package's attribute(s) from `configuration.nix` you could do:
1510
1511```nix
1512 nixpkgs.config.packageOverrides = super: {
1513 python = super.python.override {
1514 packageOverrides = python-self: python-super: {
1515 twisted = python-super.twisted.overrideAttrs (oldAttrs: {
1516 src = super.fetchPypi {
1517 pname = "twisted";
1518 version = "19.10.0";
1519 sha256 = "7394ba7f272ae722a74f3d969dcf599bc4ef093bc392038748a490f1724a515d";
1520 extension = "tar.bz2";
1521 };
1522 });
1523 };
1524 };
1525 };
1526```
1527
1528`pythonPackages.twisted` is now globally overridden.
1529All packages and also all NixOS services that reference `twisted`
1530(such as `services.buildbot-worker`) now use the new definition.
1531Note that `python-super` refers to the old package set and `python-self`
1532to the new, overridden version.
1533
1534To modify only a Python package set instead of a whole Python derivation, use
1535this snippet:
1536
1537```nix
1538 myPythonPackages = pythonPackages.override {
1539 overrides = self: super: {
1540 twisted = ...;
1541 };
1542 }
1543```
1544
1545### How to override a Python package using overlays? {#how-to-override-a-python-package-using-overlays}
1546
1547Use the following overlay template:
1548
1549```nix
1550self: super: {
1551 python = super.python.override {
1552 packageOverrides = python-self: python-super: {
1553 twisted = python-super.twisted.overrideAttrs (oldAttrs: {
1554 src = super.fetchPypi {
1555 pname = "twisted";
1556 version = "19.10.0";
1557 sha256 = "7394ba7f272ae722a74f3d969dcf599bc4ef093bc392038748a490f1724a515d";
1558 extension = "tar.bz2";
1559 };
1560 });
1561 };
1562 };
1563}
1564```
1565
1566### How to use Intel’s MKL with numpy and scipy? {#how-to-use-intels-mkl-with-numpy-and-scipy}
1567
1568MKL can be configured using an overlay. See the section "[Using overlays to
1569configure alternatives](#sec-overlays-alternatives-blas-lapack)".
1570
1571### What inputs do `setup_requires`, `install_requires` and `tests_require` map to? {#what-inputs-do-setup_requires-install_requires-and-tests_require-map-to}
1572
1573In a `setup.py` or `setup.cfg` it is common to declare dependencies:
1574
1575* `setup_requires` corresponds to `nativeBuildInputs`
1576* `install_requires` corresponds to `propagatedBuildInputs`
1577* `tests_require` corresponds to `checkInputs`
1578
1579## Contributing {#contributing}
1580
1581### Contributing guidelines {#contributing-guidelines}
1582
1583The following rules are desired to be respected:
1584
1585* Python libraries are called from `python-packages.nix` and packaged with
1586 `buildPythonPackage`. The expression of a library should be in
1587 `pkgs/development/python-modules/<name>/default.nix`.
1588* Python applications live outside of `python-packages.nix` and are packaged
1589 with `buildPythonApplication`.
1590* Make sure libraries build for all Python interpreters.
1591* By default we enable tests. Make sure the tests are found and, in the case of
1592 libraries, are passing for all interpreters. If certain tests fail they can be
1593 disabled individually. Try to avoid disabling the tests altogether. In any
1594 case, when you disable tests, leave a comment explaining why.
1595* Commit names of Python libraries should reflect that they are Python
1596 libraries, so write for example `pythonPackages.numpy: 1.11 -> 1.12`.
1597* Attribute names in `python-packages.nix` as well as `pname`s should match the
1598 library's name on PyPI, but be normalized according to [PEP
1599 0503](https://www.python.org/dev/peps/pep-0503/#normalized-names). This means
1600 that characters should be converted to lowercase and `.` and `_` should be
1601 replaced by a single `-` (foo-bar-baz instead of Foo__Bar.baz).
1602 If necessary, `pname` has to be given a different value within `fetchPypi`.
1603* Attribute names in `python-packages.nix` should be sorted alphanumerically to
1604 avoid merge conflicts and ease locating attributes.
1605
1606## Package set maintenance
1607
1608The whole Python package set has a lot of packages that do not see regular
1609updates, because they either are a very fragile component in the Python
1610ecosystem, like for example the `hypothesis` package, or packages that have
1611no maintainer, so maintenance falls back to the package set maintainers.
1612
1613### Updating packages in bulk
1614
1615There is a tool to update alot of python libraries in bulk, it exists at
1616`maintainers/scripts/update-python-libraries` with this repository.
1617
1618It can quickly update minor or major versions for all packages selected
1619and create update commits, and supports the `fetchPypi`, `fetchurl` and
1620`fetchFromGitHub` fetchers. When updating lots of packages that are
1621hosted on GitHub, exporting a `GITHUB_API_TOKEN` is highly recommended.
1622
1623Updating packages in bulk leads to lots of breakages, which is why a
1624stabilization period on the `python-unstable` branch is required.
1625
1626Once the branch is sufficiently stable it should normally be merged
1627into the `staging` branch.
1628
1629An exemplary call to update all python libraries between minor versions
1630would be:
1631
1632```ShellSession
1633$ maintainers/scripts/update-python-libraries --target minor --commit --use-pkgs-prefix pkgs/development/python-modules/**/default.nix
1634```