pyrox.dev
1# Python {#python}
2
3## User Guide
4
5### Using Python
6
7#### Overview
8
9Several versions of the Python interpreter are available on Nix, as well as a
10high amount of packages. The attribute `python3` refers to the default
11interpreter, which is currently CPython 3.8. The attribute `python` refers to
12CPython 2.7 for backwards-compatibility. It is also possible to refer to
13specific versions, e.g. `python38` refers to CPython 3.8, and `pypy` refers to
14the default PyPy interpreter.
15
16Python is used a lot, and in different ways. This affects also how it is
17packaged. In the case of Python on Nix, an important distinction is made between
18whether the package is considered primarily an application, or whether it should
19be used as a library, i.e., of primary interest are the modules in
20`site-packages` that should be importable.
21
22In the Nixpkgs tree Python applications can be found throughout, depending on
23what they do, and are called from the main package set. Python libraries,
24however, are in separate sets, with one set per interpreter version.
25
26The interpreters have several common attributes. One of these attributes is
27`pkgs`, which is a package set of Python libraries for this specific
28interpreter. E.g., the `toolz` package corresponding to the default interpreter
29is `python.pkgs.toolz`, and the CPython 3.8 version is `python38.pkgs.toolz`.
30The main package set contains aliases to these package sets, e.g.
31`pythonPackages` refers to `python.pkgs` and `python38Packages` to
32`python38.pkgs`.
33
34#### Installing Python and packages
35
36The Nix and NixOS manuals explain how packages are generally installed. In the
37case of Python and Nix, it is important to make a distinction between whether the
38package is considered an application or a library.
39
40Applications on Nix are typically installed into your user profile imperatively
41using `nix-env -i`, and on NixOS declaratively by adding the package name to
42`environment.systemPackages` in `/etc/nixos/configuration.nix`. Dependencies
43such as libraries are automatically installed and should not be installed
44explicitly.
45
46The same goes for Python applications. Python applications can be installed in
47your profile, and will be wrapped to find their exact library dependencies,
48without impacting other applications or polluting your user environment.
49
50But Python libraries you would like to use for development cannot be installed,
51at least not individually, because they won't be able to find each other
52resulting in import errors. Instead, it is possible to create an environment
53with `python.buildEnv` or `python.withPackages` where the interpreter and other
54executables are wrapped to be able to find each other and all of the modules.
55
56In the following examples we will start by creating a simple, ad-hoc environment
57with a nix-shell that has `numpy` and `toolz` in Python 3.8; then we will create
58a re-usable environment in a single-file Python script; then we will create a
59full Python environment for development with this same environment.
60
61Philosphically, this should be familiar to users who are used to a `venv` style
62of development: individual projects create their own Python environments without
63impacting the global environment or each other.
64
65#### Ad-hoc temporary Python environment with `nix-shell`
66
67The simplest way to start playing with the way nix wraps and sets up Python
68environments is with `nix-shell` at the cmdline. These environments create a
69temporary shell session with a Python and a *precise* list of packages (plus
70their runtime dependencies), with no other Python packages in the Python
71interpreter's scope.
72
73To create a Python 3.8 session with `numpy` and `toolz` available, run:
74
75```sh
76$ nix-shell -p 'python38.withPackages(ps: with ps; [ numpy toolz ])'
77```
78
79By default `nix-shell` will start a `bash` session with this interpreter in our
80`PATH`, so if we then run:
81
82```Python console
83[nix-shell:~/src/nixpkgs]$ python3
84Python 3.8.1 (default, Dec 18 2019, 19:06:26)
85[GCC 9.2.0] on linux
86Type "help", "copyright", "credits" or "license" for more information.
87>>> import numpy; import toolz
88```
89
90Note that no other modules are in scope, even if they were imperatively
91installed into our user environment as a dependency of a Python application:
92
93```Python console
94>>> import requests
95Traceback (most recent call last):
96 File "<stdin>", line 1, in <module>
97ModuleNotFoundError: No module named 'requests'
98```
99
100We can add as many additional modules onto the `nix-shell` as we need, and we
101will still get 1 wrapped Python interpreter. We can start the interpreter
102directly like so:
103
104```sh
105$ nix-shell -p 'python38.withPackages(ps: with ps; [ numpy toolz requests ])' --run python3
106these derivations will be built:
107 /nix/store/xbdsrqrsfa1yva5s7pzsra8k08gxlbz1-python3-3.8.1-env.drv
108building '/nix/store/xbdsrqrsfa1yva5s7pzsra8k08gxlbz1-python3-3.8.1-env.drv'...
109created 277 symlinks in user environment
110Python 3.8.1 (default, Dec 18 2019, 19:06:26)
111[GCC 9.2.0] on linux
112Type "help", "copyright", "credits" or "license" for more information.
113>>> import requests
114>>>
115```
116
117Notice that this time it built a new Python environment, which now includes
118`requests`. Building an environment just creates wrapper scripts that expose the
119selected dependencies to the interpreter while re-using the actual modules. This
120means if any other env has installed `requests` or `numpy` in a different
121context, we don't need to recompile them -- we just recompile the wrapper script
122that sets up an interpreter pointing to them. This matters much more for "big"
123modules like `pytorch` or `tensorflow`.
124
125Module names usually match their names on [pypi.org](https://pypi.org/), but
126you can use the [Nixpkgs search website](https://nixos.org/nixos/packages.html)
127to find them as well (along with non-python packages).
128
129At this point we can create throwaway experimental Python environments with
130arbitrary dependencies. This is a good way to get a feel for how the Python
131interpreter and dependencies work in Nix and NixOS, but to do some actual
132development, we'll want to make it a bit more persistent.
133
134##### Running Python scripts and using `nix-shell` as shebang
135
136Sometimes, we have a script whose header looks like this:
137
138```python
139#!/usr/bin/env python3
140import numpy as np
141a = np.array([1,2])
142b = np.array([3,4])
143print(f"The dot product of {a} and {b} is: {np.dot(a, b)}")
144```
145
146Executing this script requires a `python3` that has `numpy`. Using what we learned
147in the previous section, we could startup a shell and just run it like so:
148
149```ShellSesssion
150$ nix-shell -p 'python38.withPackages(ps: with ps; [ numpy ])' --run 'python3 foo.py'
151The dot product of [1 2] and [3 4] is: 11
152```
153
154But if we maintain the script ourselves, and if there are more dependencies, it
155may be nice to encode those dependencies in source to make the script re-usable
156without that bit of knowledge. That can be done by using `nix-shell` as a
157[shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)), like so:
158
159```python
160#!/usr/bin/env nix-shell
161#!nix-shell -i python3 -p "python3.withPackages(ps: [ ps.numpy ])"
162import numpy as np
163a = np.array([1,2])
164b = np.array([3,4])
165print(f"The dot product of {a} and {b} is: {np.dot(a, b)}")
166```
167
168Then we simply execute it, without requiring any environment setup at all!
169
170```sh
171$ ./foo.py
172The dot product of [1 2] and [3 4] is: 11
173```
174
175If the dependencies are not available on the host where `foo.py` is executed, it
176will build or download them from a Nix binary cache prior to starting up, prior
177that it is executed on a machine with a multi-user nix installation.
178
179This provides a way to ship a self bootstrapping Python script, akin to a
180statically linked binary, where it can be run on any machine (provided nix is
181installed) without having to assume that `numpy` is installed globally on the
182system.
183
184By default it is pulling the import checkout of Nixpkgs itself from our nix
185channel, which is nice as it cache aligns with our other package builds, but we
186can make it fully reproducible by pinning the `nixpkgs` import:
187
188```python
189#!/usr/bin/env nix-shell
190#!nix-shell -i python3 -p "python3.withPackages(ps: [ ps.numpy ])"
191#!nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/d373d80b1207d52621961b16aa4a3438e4f98167.tar.gz
192import numpy as np
193a = np.array([1,2])
194b = np.array([3,4])
195print(f"The dot product of {a} and {b} is: {np.dot(a, b)}")
196```
197
198This will execute with the exact same versions of Python 3.8, numpy, and system
199dependencies a year from now as it does today, because it will always use
200exactly git commit `d373d80b1207d52621961b16aa4a3438e4f98167` of Nixpkgs for all
201of the package versions.
202
203This is also a great way to ensure the script executes identically on different
204servers.
205
206##### Load environment from `.nix` expression
207
208We've now seen how to create an ad-hoc temporary shell session, and how to
209create a single script with Python dependencies, but in the course of normal
210development we're usually working in an entire package repository.
211
212As explained in the Nix manual, `nix-shell` can also load an expression from a
213`.nix` file. Say we want to have Python 3.8, `numpy` and `toolz`, like before,
214in an environment. We can add a `shell.nix` file describing our dependencies:
215
216```nix
217with import <nixpkgs> {};
218(python38.withPackages (ps: [ps.numpy ps.toolz])).env
219```
220
221And then at the command line, just typing `nix-shell` produces the same
222environment as before. In a normal project, we'll likely have many more
223dependencies; this can provide a way for developers to share the environments
224with each other and with CI builders.
225
226What's happening here?
227
2281. We begin with importing the Nix Packages collections. `import <nixpkgs>`
229 imports the `<nixpkgs>` function, `{}` calls it and the `with` statement
230 brings all attributes of `nixpkgs` in the local scope. These attributes form
231 the main package set.
2322. Then we create a Python 3.8 environment with the `withPackages` function, as before.
2333. The `withPackages` function expects us to provide a function as an argument
234 that takes the set of all Python packages and returns a list of packages to
235 include in the environment. Here, we select the packages `numpy` and `toolz`
236 from the package set.
237
238To combine this with `mkShell` you can:
239
240```nix
241with import <nixpkgs> {};
242let
243 pythonEnv = python38.withPackages (ps: [
244 ps.numpy
245 ps.toolz
246 ]);
247in mkShell {
248 packages = [
249 pythonEnv
250
251 black
252 mypy
253
254 libffi
255 openssl
256 ];
257}
258```
259
260This will create a unified environment that has not just our Python interpreter
261and its Python dependencies, but also tools like `black` or `mypy` and libraries
262like `libffi` the `openssl` in scope. This is generic and can span any number of
263tools or languages across the Nixpkgs ecosystem.
264
265##### Installing environments globally on the system
266
267Up to now, we've been creating environments scoped to an ad-hoc shell session,
268or a single script, or a single project. This is generally advisable, as it
269avoids pollution across contexts.
270
271However, sometimes we know we will often want a Python with some basic packages,
272and want this available without having to enter into a shell or build context.
273This can be useful to have things like vim/emacs editors and plugins or shell
274tools "just work" without having to set them up, or when running other software
275that expects packages to be installed globally.
276
277To create your own custom environment, create a file in `~/.config/nixpkgs/overlays/`
278that looks like this:
279
280```nix
281# ~/.config/nixpkgs/overlays/myEnv.nix
282self: super: {
283 myEnv = super.buildEnv {
284 name = "myEnv";
285 paths = [
286 # A Python 3 interpreter with some packages
287 (self.python3.withPackages (
288 ps: with ps; [
289 pyflakes
290 pytest
291 python-language-server
292 ]
293 ))
294
295 # Some other packages we'd like as part of this env
296 self.mypy
297 self.black
298 self.ripgrep
299 self.tmux
300 ];
301 };
302}
303```
304
305You can then build and install this to your profile with:
306
307```sh
308nix-env -iA myEnv
309```
310
311One limitation of this is that you can only have 1 Python env installed
312globally, since they conflict on the `python` to load out of your `PATH`.
313
314If you get a conflict or prefer to keep the setup clean, you can have `nix-env`
315atomically *uninstall* all other imperatively installed packages and replace
316your profile with just `myEnv` by using the `--replace` flag.
317
318##### Environment defined in `/etc/nixos/configuration.nix`
319
320For the sake of completeness, here's how to install the environment system-wide
321on NixOS.
322
323```nix
324{ # ...
325
326 environment.systemPackages = with pkgs; [
327 (python38.withPackages(ps: with ps; [ numpy toolz ]))
328 ];
329}
330```
331
332### Developing with Python
333
334Above, we were mostly just focused on use cases and what to do to get started
335creating working Python environments in nix.
336
337Now that you know the basics to be up and running, it is time to take a step
338back and take a deeper look at how Python packages are packaged on Nix. Then,
339we will look at how you can use development mode with your code.
340
341#### Python library packages in Nixpkgs
342
343With Nix all packages are built by functions. The main function in Nix for
344building Python libraries is `buildPythonPackage`. Let's see how we can build the
345`toolz` package.
346
347```nix
348{ lib, buildPythonPackage, fetchPypi }:
349
350buildPythonPackage rec {
351 pname = "toolz";
352 version = "0.10.0";
353
354 src = fetchPypi {
355 inherit pname version;
356 sha256 = "08fdd5ef7c96480ad11c12d472de21acd32359996f69a5259299b540feba4560";
357 };
358
359 doCheck = false;
360
361 meta = with lib; {
362 homepage = "https://github.com/pytoolz/toolz";
363 description = "List processing tools and functional utilities";
364 license = licenses.bsd3;
365 maintainers = with maintainers; [ fridh ];
366 };
367}
368```
369
370What happens here? The function `buildPythonPackage` is called and as argument
371it accepts a set. In this case the set is a recursive set, `rec`. One of the
372arguments is the name of the package, which consists of a basename (generally
373following the name on PyPi) and a version. Another argument, `src` specifies the
374source, which in this case is fetched from PyPI using the helper function
375`fetchPypi`. The argument `doCheck` is used to set whether tests should be run
376when building the package. Furthermore, we specify some (optional) meta
377information. The output of the function is a derivation.
378
379An expression for `toolz` can be found in the Nixpkgs repository. As explained
380in the introduction of this Python section, a derivation of `toolz` is available
381for each interpreter version, e.g. `python38.pkgs.toolz` refers to the `toolz`
382derivation corresponding to the CPython 3.8 interpreter.
383
384The above example works when you're directly working on
385`pkgs/top-level/python-packages.nix` in the Nixpkgs repository. Often though,
386you will want to test a Nix expression outside of the Nixpkgs tree.
387
388The following expression creates a derivation for the `toolz` package,
389and adds it along with a `numpy` package to a Python environment.
390
391```nix
392with import <nixpkgs> {};
393
394( let
395 my_toolz = python38.pkgs.buildPythonPackage rec {
396 pname = "toolz";
397 version = "0.10.0";
398
399 src = python38.pkgs.fetchPypi {
400 inherit pname version;
401 sha256 = "08fdd5ef7c96480ad11c12d472de21acd32359996f69a5259299b540feba4560";
402 };
403
404 doCheck = false;
405
406 meta = {
407 homepage = "https://github.com/pytoolz/toolz/";
408 description = "List processing tools and functional utilities";
409 };
410 };
411
412 in python38.withPackages (ps: [ps.numpy my_toolz])
413).env
414```
415
416Executing `nix-shell` will result in an environment in which you can use
417Python 3.8 and the `toolz` package. As you can see we had to explicitly mention
418for which Python version we want to build a package.
419
420So, what did we do here? Well, we took the Nix expression that we used earlier
421to build a Python environment, and said that we wanted to include our own
422version of `toolz`, named `my_toolz`. To introduce our own package in the scope
423of `withPackages` we used a `let` expression. You can see that we used
424`ps.numpy` to select numpy from the nixpkgs package set (`ps`). We did not take
425`toolz` from the Nixpkgs package set this time, but instead took our own version
426that we introduced with the `let` expression.
427
428#### Handling dependencies
429
430Our example, `toolz`, does not have any dependencies on other Python packages or
431system libraries. According to the manual, `buildPythonPackage` uses the
432arguments `buildInputs` and `propagatedBuildInputs` to specify dependencies. If
433something is exclusively a build-time dependency, then the dependency should be
434included in `buildInputs`, but if it is (also) a runtime dependency, then it
435should be added to `propagatedBuildInputs`. Test dependencies are considered
436build-time dependencies and passed to `checkInputs`.
437
438The following example shows which arguments are given to `buildPythonPackage` in
439order to build [`datashape`](https://github.com/blaze/datashape).
440
441```nix
442{ lib, buildPythonPackage, fetchPypi, numpy, multipledispatch, dateutil, pytest }:
443
444buildPythonPackage rec {
445 pname = "datashape";
446 version = "0.4.7";
447
448 src = fetchPypi {
449 inherit pname version;
450 sha256 = "14b2ef766d4c9652ab813182e866f493475e65e558bed0822e38bf07bba1a278";
451 };
452
453 checkInputs = [ pytest ];
454 propagatedBuildInputs = [ numpy multipledispatch dateutil ];
455
456 meta = with lib; {
457 homepage = "https://github.com/ContinuumIO/datashape";
458 description = "A data description language";
459 license = licenses.bsd2;
460 maintainers = with maintainers; [ fridh ];
461 };
462}
463```
464
465We can see several runtime dependencies, `numpy`, `multipledispatch`, and
466`dateutil`. Furthermore, we have one `checkInputs`, i.e. `pytest`. `pytest` is a
467test runner and is only used during the `checkPhase` and is therefore not added
468to `propagatedBuildInputs`.
469
470In the previous case we had only dependencies on other Python packages to consider.
471Occasionally you have also system libraries to consider. E.g., `lxml` provides
472Python bindings to `libxml2` and `libxslt`. These libraries are only required
473when building the bindings and are therefore added as `buildInputs`.
474
475```nix
476{ lib, pkgs, buildPythonPackage, fetchPypi }:
477
478buildPythonPackage rec {
479 pname = "lxml";
480 version = "3.4.4";
481
482 src = fetchPypi {
483 inherit pname version;
484 sha256 = "16a0fa97hym9ysdk3rmqz32xdjqmy4w34ld3rm3jf5viqjx65lxk";
485 };
486
487 buildInputs = [ pkgs.libxml2 pkgs.libxslt ];
488
489 meta = with lib; {
490 description = "Pythonic binding for the libxml2 and libxslt libraries";
491 homepage = "https://lxml.de";
492 license = licenses.bsd3;
493 maintainers = with maintainers; [ sjourdois ];
494 };
495}
496```
497
498In this example `lxml` and Nix are able to work out exactly where the relevant
499files of the dependencies are. This is not always the case.
500
501The example below shows bindings to The Fastest Fourier Transform in the West,
502commonly known as FFTW. On Nix we have separate packages of FFTW for the
503different types of floats (`"single"`, `"double"`, `"long-double"`). The
504bindings need all three types, and therefore we add all three as `buildInputs`.
505The bindings don't expect to find each of them in a different folder, and
506therefore we have to set `LDFLAGS` and `CFLAGS`.
507
508```nix
509{ lib, pkgs, buildPythonPackage, fetchPypi, numpy, scipy }:
510
511buildPythonPackage rec {
512 pname = "pyFFTW";
513 version = "0.9.2";
514
515 src = fetchPypi {
516 inherit pname version;
517 sha256 = "f6bbb6afa93085409ab24885a1a3cdb8909f095a142f4d49e346f2bd1b789074";
518 };
519
520 buildInputs = [ pkgs.fftw pkgs.fftwFloat pkgs.fftwLongDouble];
521
522 propagatedBuildInputs = [ numpy scipy ];
523
524 # Tests cannot import pyfftw. pyfftw works fine though.
525 doCheck = false;
526
527 preConfigure = ''
528 export LDFLAGS="-L${pkgs.fftw.dev}/lib -L${pkgs.fftwFloat.out}/lib -L${pkgs.fftwLongDouble.out}/lib"
529 export CFLAGS="-I${pkgs.fftw.dev}/include -I${pkgs.fftwFloat.dev}/include -I${pkgs.fftwLongDouble.dev}/include"
530 '';
531
532 meta = with lib; {
533 description = "A pythonic wrapper around FFTW, the FFT library, presenting a unified interface for all the supported transforms";
534 homepage = "http://hgomersall.github.com/pyFFTW";
535 license = with licenses; [ bsd2 bsd3 ];
536 maintainers = with maintainers; [ fridh ];
537 };
538}
539```
540Note also the line `doCheck = false;`, we explicitly disabled running the test-suite.
541
542#### Testing Python Packages
543
544It is highly encouraged to have testing as part of the package build. This
545helps to avoid situations where the package was able to build and install,
546but is not usable at runtime. Currently, all packages will use the `test`
547command provided by the setup.py (i.e. `python setup.py test`). However,
548this is currently deprecated https://github.com/pypa/setuptools/pull/1878
549and your package should provide its own checkPhase.
550
551*NOTE:* The `checkPhase` for python maps to the `installCheckPhase` on a
552normal derivation. This is due to many python packages not behaving well
553to the pre-installed version of the package. Version info, and natively
554compiled extensions generally only exist in the install directory, and
555thus can cause issues when a test suite asserts on that behavior.
556
557*NOTE:* Tests should only be disabled if they don't agree with nix
558(e.g. external dependencies, network access, flakey tests), however,
559as many tests should be enabled as possible. Failing tests can still be
560a good indication that the package is not in a valid state.
561
562#### Using pytest
563
564Pytest is the most common test runner for python repositories. A trivial
565test run would be:
566```
567 checkInputs = [ pytest ];
568 checkPhase = "pytest";
569```
570
571However, many repositories' test suites do not translate well to nix's build
572sandbox, and will generally need many tests to be disabled.
573
574To filter tests using pytest, one can do the following:
575```
576 checkInputs = [ pytest ];
577 # avoid tests which need additional data or touch network
578 checkPhase = ''
579 pytest tests/ --ignore=tests/integration -k 'not download and not update'
580 '';
581```
582
583`--ignore` will tell pytest to ignore that file or directory from being
584collected as part of a test run. This is useful is a file uses a package
585which is not available in nixpkgs, thus skipping that test file is much
586easier than having to create a new package.
587
588`-k` is used to define a predicate for test names. In this example, we are
589filtering out tests which contain `download` or `update` in their test case name.
590Only one `-k` argument is allows, and thus a long predicate should be concatenated
591with "\" and wrapped to the next line.
592
593*NOTE:* In pytest==6.0.1, the use of "\" to continue a line (e.g. `-k 'not download \'`) has
594been removed, in this case, it's recommended to use `pytestCheckHook`.
595
596#### Using pytestCheckHook
597
598`pytestCheckHook` is a convenient hook which will substitute the setuptools
599`test` command for a checkPhase which runs `pytest`. This is also beneficial
600when a package may need many items disabled to run the test suite.
601
602Using the example above, the analagous pytestCheckHook usage would be:
603```
604 checkInputs = [ pytestCheckHook ];
605
606 # requires additional data
607 pytestFlagsArray = [ "tests/" "--ignore=tests/integration" ];
608
609 disabledTests = [
610 # touches network
611 "download"
612 "update"
613 ];
614
615 disabledTestPaths = [
616 "tests/test_failing.py"
617 ];
618```
619
620This is expecially useful when tests need to be conditionallydisabled,
621for example:
622
623```
624 disabledTests = [
625 # touches network
626 "download"
627 "update"
628 ] ++ lib.optionals (pythonAtLeast "3.8") [
629 # broken due to python3.8 async changes
630 "async"
631 ] ++ lib.optionals stdenv.isDarwin [
632 # can fail when building with other packages
633 "socket"
634 ];
635```
636Trying to concatenate the related strings to disable tests in a regular checkPhase
637would be much harder to read. This also enables us to comment on why specific tests
638are disabled.
639
640#### Using pythonImportsCheck
641
642Although unit tests are highly prefered to validate correctness of a package, not
643all packages have test suites that can be ran easily, and some have none at all.
644To help ensure the package still works, `pythonImportsCheck` can attempt to import
645the listed modules.
646
647```
648 pythonImportsCheck = [ "requests" "urllib" ];
649```
650roughly translates to:
651```
652 postCheck = ''
653 PYTHONPATH=$out/${python.sitePackages}:$PYTHONPATH
654 python -c "import requests; import urllib"
655 '';
656```
657However, this is done in it's own phase, and not dependent on whether `doCheck = true;`
658
659This can also be useful in verifying that the package doesn't assume commonly
660present packages (e.g. `setuptools`)
661
662### Develop local package
663
664As a Python developer you're likely aware of [development mode](http://setuptools.readthedocs.io/en/latest/setuptools.html#development-mode)
665(`python setup.py develop`); instead of installing the package this command
666creates a special link to the project code. That way, you can run updated code
667without having to reinstall after each and every change you make. Development
668mode is also available. Let's see how you can use it.
669
670In the previous Nix expression the source was fetched from an url. We can also
671refer to a local source instead using `src = ./path/to/source/tree;`
672
673If we create a `shell.nix` file which calls `buildPythonPackage`, and if `src`
674is a local source, and if the local source has a `setup.py`, then development
675mode is activated.
676
677In the following example we create a simple environment that has a Python 3.8
678version of our package in it, as well as its dependencies and other packages we
679like to have in the environment, all specified with `propagatedBuildInputs`.
680Indeed, we can just add any package we like to have in our environment to
681`propagatedBuildInputs`.
682
683```nix
684with import <nixpkgs> {};
685with python38Packages;
686
687buildPythonPackage rec {
688 name = "mypackage";
689 src = ./path/to/package/source;
690 propagatedBuildInputs = [ pytest numpy pkgs.libsndfile ];
691}
692```
693
694It is important to note that due to how development mode is implemented on Nix
695it is not possible to have multiple packages simultaneously in development mode.
696
697### Organising your packages
698
699So far we discussed how you can use Python on Nix, and how you can develop with
700it. We've looked at how you write expressions to package Python packages, and we
701looked at how you can create environments in which specified packages are
702available.
703
704At some point you'll likely have multiple packages which you would
705like to be able to use in different projects. In order to minimise unnecessary
706duplication we now look at how you can maintain a repository with your
707own packages. The important functions here are `import` and `callPackage`.
708
709### Including a derivation using `callPackage`
710
711Earlier we created a Python environment using `withPackages`, and included the
712`toolz` package via a `let` expression.
713Let's split the package definition from the environment definition.
714
715We first create a function that builds `toolz` in `~/path/to/toolz/release.nix`
716
717```nix
718{ lib, buildPythonPackage }:
719
720buildPythonPackage rec {
721 pname = "toolz";
722 version = "0.10.0";
723
724 src = fetchPypi {
725 inherit pname version;
726 sha256 = "08fdd5ef7c96480ad11c12d472de21acd32359996f69a5259299b540feba4560";
727 };
728
729 meta = with lib; {
730 homepage = "https://github.com/pytoolz/toolz/";
731 description = "List processing tools and functional utilities";
732 license = licenses.bsd3;
733 maintainers = with maintainers; [ fridh ];
734 };
735}
736```
737
738It takes an argument `buildPythonPackage`. We now call this function using
739`callPackage` in the definition of our environment
740
741```nix
742with import <nixpkgs> {};
743
744( let
745 toolz = callPackage /path/to/toolz/release.nix {
746 buildPythonPackage = python38Packages.buildPythonPackage;
747 };
748 in python38.withPackages (ps: [ ps.numpy toolz ])
749).env
750```
751
752Important to remember is that the Python version for which the package is made
753depends on the `python` derivation that is passed to `buildPythonPackage`. Nix
754tries to automatically pass arguments when possible, which is why generally you
755don't explicitly define which `python` derivation should be used. In the above
756example we use `buildPythonPackage` that is part of the set `python38Packages`,
757and in this case the `python38` interpreter is automatically used.
758
759## Reference
760
761### Interpreters
762
763Versions 2.7, 3.6, 3.7, 3.8 and 3.9 of the CPython interpreter are available as
764respectively `python27`, `python36`, `python37`, `python38` and `python39`. The
765aliases `python2` and `python3` correspond to respectively `python27` and
766`python39`. The attribute `python` maps to `python2`. The PyPy interpreters
767compatible with Python 2.7 and 3 are available as `pypy27` and `pypy3`, with
768aliases `pypy2` mapping to `pypy27` and `pypy` mapping to `pypy2`. The Nix
769expressions for the interpreters can be found in
770`pkgs/development/interpreters/python`.
771
772All packages depending on any Python interpreter get appended
773`out/{python.sitePackages}` to `$PYTHONPATH` if such directory
774exists.
775
776#### Missing `tkinter` module standard library
777
778To reduce closure size the `Tkinter`/`tkinter` is available as a separate package, `pythonPackages.tkinter`.
779
780#### Attributes on interpreters packages
781
782Each interpreter has the following attributes:
783
784- `libPrefix`. Name of the folder in `${python}/lib/` for corresponding interpreter.
785- `interpreter`. Alias for `${python}/bin/${executable}`.
786- `buildEnv`. Function to build python interpreter environments with extra packages bundled together. See section *python.buildEnv function* for usage and documentation.
787- `withPackages`. Simpler interface to `buildEnv`. See section *python.withPackages function* for usage and documentation.
788- `sitePackages`. Alias for `lib/${libPrefix}/site-packages`.
789- `executable`. Name of the interpreter executable, e.g. `python3.8`.
790- `pkgs`. Set of Python packages for that specific interpreter. The package set can be modified by overriding the interpreter and passing `packageOverrides`.
791
792### Optimizations
793
794The Python interpreters are by default not build with optimizations enabled, because
795the builds are in that case not reproducible. To enable optimizations, override the
796interpreter of interest, e.g using
797
798```
799let
800 pkgs = import ./. {};
801 mypython = pkgs.python3.override {
802 enableOptimizations = true;
803 reproducibleBuild = false;
804 self = mypython;
805 };
806in mypython
807```
808
809### Building packages and applications
810
811Python libraries and applications that use `setuptools` or
812`distutils` are typically built with respectively the `buildPythonPackage` and
813`buildPythonApplication` functions. These two functions also support installing a `wheel`.
814
815All Python packages reside in `pkgs/top-level/python-packages.nix` and all
816applications elsewhere. In case a package is used as both a library and an
817application, then the package should be in `pkgs/top-level/python-packages.nix`
818since only those packages are made available for all interpreter versions. The
819preferred location for library expressions is in
820`pkgs/development/python-modules`. It is important that these packages are
821called from `pkgs/top-level/python-packages.nix` and not elsewhere, to guarantee
822the right version of the package is built.
823
824Based on the packages defined in `pkgs/top-level/python-packages.nix` an
825attribute set is created for each available Python interpreter. The available
826sets are
827
828* `pkgs.python27Packages`
829* `pkgs.python36Packages`
830* `pkgs.python37Packages`
831* `pkgs.python38Packages`
832* `pkgs.python39Packages`
833* `pkgs.pypyPackages`
834
835and the aliases
836
837* `pkgs.python2Packages` pointing to `pkgs.python27Packages`
838* `pkgs.python3Packages` pointing to `pkgs.python38Packages`
839* `pkgs.pythonPackages` pointing to `pkgs.python2Packages`
840
841#### `buildPythonPackage` function
842
843The `buildPythonPackage` function is implemented in
844`pkgs/development/interpreters/python/mk-python-derivation`
845using setup hooks.
846
847The following is an example:
848
849```nix
850{ lib, buildPythonPackage, fetchPypi, hypothesis, setuptools_scm, attrs, py, setuptools, six, pluggy }:
851
852buildPythonPackage rec {
853 pname = "pytest";
854 version = "3.3.1";
855
856 src = fetchPypi {
857 inherit pname version;
858 sha256 = "cf8436dc59d8695346fcd3ab296de46425ecab00d64096cebe79fb51ecb2eb93";
859 };
860
861 postPatch = ''
862 # don't test bash builtins
863 rm testing/test_argcomplete.py
864 '';
865
866 checkInputs = [ hypothesis ];
867 nativeBuildInputs = [ setuptools_scm ];
868 propagatedBuildInputs = [ attrs py setuptools six pluggy ];
869
870 meta = with lib; {
871 maintainers = with maintainers; [ domenkozar lovek323 madjar lsix ];
872 description = "Framework for writing tests";
873 };
874}
875```
876
877The `buildPythonPackage` mainly does four things:
878
879* In the `buildPhase`, it calls `${python.interpreter} setup.py bdist_wheel` to
880 build a wheel binary zipfile.
881* In the `installPhase`, it installs the wheel file using `pip install *.whl`.
882* In the `postFixup` phase, the `wrapPythonPrograms` bash function is called to
883 wrap all programs in the `$out/bin/*` directory to include `$PATH`
884 environment variable and add dependent libraries to script's `sys.path`.
885* In the `installCheck` phase, `${python.interpreter} setup.py test` is ran.
886
887By default tests are run because `doCheck = true`. Test dependencies, like
888e.g. the test runner, should be added to `checkInputs`.
889
890By default `meta.platforms` is set to the same value
891as the interpreter unless overridden otherwise.
892
893##### `buildPythonPackage` parameters
894
895All parameters from `stdenv.mkDerivation` function are still supported. The
896following are specific to `buildPythonPackage`:
897
898* `catchConflicts ? true`: If `true`, abort package build if a package name
899 appears more than once in dependency tree. Default is `true`.
900* `disabled` ? false: If `true`, package is not built for the particular Python
901 interpreter version.
902* `dontWrapPythonPrograms ? false`: Skip wrapping of Python programs.
903* `permitUserSite ? false`: Skip setting the `PYTHONNOUSERSITE` environment
904 variable in wrapped programs.
905* `format ? "setuptools"`: Format of the source. Valid options are
906 `"setuptools"`, `"pyproject"`, `"flit"`, `"wheel"`, and `"other"`.
907 `"setuptools"` is for when the source has a `setup.py` and `setuptools` is
908 used to build a wheel, `flit`, in case `flit` should be used to build a wheel,
909 and `wheel` in case a wheel is provided. Use `other` when a custom
910 `buildPhase` and/or `installPhase` is needed.
911* `makeWrapperArgs ? []`: A list of strings. Arguments to be passed to
912 `makeWrapper`, which wraps generated binaries. By default, the arguments to
913 `makeWrapper` set `PATH` and `PYTHONPATH` environment variables before calling
914 the binary. Additional arguments here can allow a developer to set environment
915 variables which will be available when the binary is run. For example,
916 `makeWrapperArgs = ["--set FOO BAR" "--set BAZ QUX"]`.
917* `namePrefix`: Prepends text to `${name}` parameter. In case of libraries, this
918 defaults to `"python3.8-"` for Python 3.8, etc., and in case of applications
919 to `""`.
920* `pipInstallFlags ? []`: A list of strings. Arguments to be passed to `pip
921 install`. To pass options to `python setup.py install`, use
922 `--install-option`. E.g., `pipInstallFlags=["--install-option='--cpp_implementation'"]`.
923* `pythonPath ? []`: List of packages to be added into `$PYTHONPATH`. Packages
924 in `pythonPath` are not propagated (contrary to `propagatedBuildInputs`).
925* `preShellHook`: Hook to execute commands before `shellHook`.
926* `postShellHook`: Hook to execute commands after `shellHook`.
927* `removeBinByteCode ? true`: Remove bytecode from `/bin`. Bytecode is only
928 created when the filenames end with `.py`.
929* `setupPyGlobalFlags ? []`: List of flags passed to `setup.py` command.
930* `setupPyBuildFlags ? []`: List of flags passed to `setup.py build_ext` command.
931
932The `stdenv.mkDerivation` function accepts various parameters for describing
933build inputs (see "Specifying dependencies"). The following are of special
934interest for Python packages, either because these are primarily used, or
935because their behaviour is different:
936
937* `nativeBuildInputs ? []`: Build-time only dependencies. Typically executables
938 as well as the items listed in `setup_requires`.
939* `buildInputs ? []`: Build and/or run-time dependencies that need to be
940 compiled for the host machine. Typically non-Python libraries which are being
941 linked.
942* `checkInputs ? []`: Dependencies needed for running the `checkPhase`. These
943 are added to `nativeBuildInputs` when `doCheck = true`. Items listed in
944 `tests_require` go here.
945* `propagatedBuildInputs ? []`: Aside from propagating dependencies,
946 `buildPythonPackage` also injects code into and wraps executables with the
947 paths included in this list. Items listed in `install_requires` go here.
948
949##### Overriding Python packages
950
951The `buildPythonPackage` function has a `overridePythonAttrs` method that can be
952used to override the package. In the following example we create an environment
953where we have the `blaze` package using an older version of `pandas`. We
954override first the Python interpreter and pass `packageOverrides` which contains
955the overrides for packages in the package set.
956
957```nix
958with import <nixpkgs> {};
959
960(let
961 python = let
962 packageOverrides = self: super: {
963 pandas = super.pandas.overridePythonAttrs(old: rec {
964 version = "0.19.1";
965 src = super.fetchPypi {
966 pname = "pandas";
967 inherit version;
968 sha256 = "08blshqj9zj1wyjhhw3kl2vas75vhhicvv72flvf1z3jvapgw295";
969 };
970 });
971 };
972 in pkgs.python3.override {inherit packageOverrides; self = python;};
973
974in python.withPackages(ps: [ps.blaze])).env
975```
976
977#### `buildPythonApplication` function
978
979The `buildPythonApplication` function is practically the same as
980`buildPythonPackage`. The main purpose of this function is to build a Python
981package where one is interested only in the executables, and not importable
982modules. For that reason, when adding this package to a `python.buildEnv`, the
983modules won't be made available.
984
985Another difference is that `buildPythonPackage` by default prefixes the names of
986the packages with the version of the interpreter. Because this is irrelevant for
987applications, the prefix is omitted.
988
989When packaging a Python application with `buildPythonApplication`, it should be
990called with `callPackage` and passed `python` or `pythonPackages` (possibly
991specifying an interpreter version), like this:
992
993```nix
994{ lib, python3Packages }:
995
996python3Packages.buildPythonApplication rec {
997 pname = "luigi";
998 version = "2.7.9";
999
1000 src = python3Packages.fetchPypi {
1001 inherit pname version;
1002 sha256 = "035w8gqql36zlan0xjrzz9j4lh9hs0qrsgnbyw07qs7lnkvbdv9x";
1003 };
1004
1005 propagatedBuildInputs = with python3Packages; [ tornado_4 python-daemon ];
1006
1007 meta = with lib; {
1008 ...
1009 };
1010}
1011```
1012
1013This is then added to `all-packages.nix` just as any other application would be.
1014
1015```nix
1016luigi = callPackage ../applications/networking/cluster/luigi { };
1017```
1018
1019Since the package is an application, a consumer doesn't need to care about
1020Python versions or modules, which is why they don't go in `pythonPackages`.
1021
1022#### `toPythonApplication` function
1023
1024A distinction is made between applications and libraries, however, sometimes a
1025package is used as both. In this case the package is added as a library to
1026`python-packages.nix` and as an application to `all-packages.nix`. To reduce
1027duplication the `toPythonApplication` can be used to convert a library to an
1028application.
1029
1030The Nix expression shall use `buildPythonPackage` and be called from
1031`python-packages.nix`. A reference shall be created from `all-packages.nix` to
1032the attribute in `python-packages.nix`, and the `toPythonApplication` shall be
1033applied to the reference:
1034```nix
1035youtube-dl = with pythonPackages; toPythonApplication youtube-dl;
1036```
1037
1038#### `toPythonModule` function
1039
1040In some cases, such as bindings, a package is created using
1041`stdenv.mkDerivation` and added as attribute in `all-packages.nix`. The Python
1042bindings should be made available from `python-packages.nix`. The
1043`toPythonModule` function takes a derivation and makes certain Python-specific
1044modifications.
1045
1046```nix
1047opencv = toPythonModule (pkgs.opencv.override {
1048 enablePython = true;
1049 pythonPackages = self;
1050});
1051```
1052
1053Do pay attention to passing in the right Python version!
1054
1055#### `python.buildEnv` function
1056
1057Python environments can be created using the low-level `pkgs.buildEnv` function.
1058This example shows how to create an environment that has the Pyramid Web Framework.
1059Saving the following as `default.nix`
1060
1061```nix
1062with import <nixpkgs> {};
1063
1064python.buildEnv.override {
1065 extraLibs = [ pythonPackages.pyramid ];
1066 ignoreCollisions = true;
1067}
1068```
1069
1070and running `nix-build` will create
1071
1072```
1073/nix/store/cf1xhjwzmdki7fasgr4kz6di72ykicl5-python-2.7.8-env
1074```
1075
1076with wrapped binaries in `bin/`.
1077
1078You can also use the `env` attribute to create local environments with needed
1079packages installed. This is somewhat comparable to `virtualenv`. For example,
1080running `nix-shell` with the following `shell.nix`
1081
1082```nix
1083with import <nixpkgs> {};
1084
1085(python3.buildEnv.override {
1086 extraLibs = with python3Packages; [ numpy requests ];
1087}).env
1088```
1089
1090will drop you into a shell where Python will have the
1091specified packages in its path.
1092
1093
1094##### `python.buildEnv` arguments
1095
1096* `extraLibs`: List of packages installed inside the environment.
1097* `postBuild`: Shell command executed after the build of environment.
1098* `ignoreCollisions`: Ignore file collisions inside the environment (default is `false`).
1099* `permitUserSite`: Skip setting the `PYTHONNOUSERSITE` environment variable in
1100 wrapped binaries in the environment.
1101
1102#### `python.withPackages` function
1103
1104The `python.withPackages` function provides a simpler interface to the `python.buildEnv` functionality.
1105It takes a function as an argument that is passed the set of python packages and returns the list
1106of the packages to be included in the environment. Using the `withPackages` function, the previous
1107example for the Pyramid Web Framework environment can be written like this:
1108
1109```nix
1110with import <nixpkgs> {};
1111
1112python.withPackages (ps: [ps.pyramid])
1113```
1114
1115`withPackages` passes the correct package set for the specific interpreter
1116version as an argument to the function. In the above example, `ps` equals
1117`pythonPackages`. But you can also easily switch to using python3:
1118
1119```nix
1120with import <nixpkgs> {};
1121
1122python3.withPackages (ps: [ps.pyramid])
1123```
1124
1125Now, `ps` is set to `python3Packages`, matching the version of the interpreter.
1126
1127As `python.withPackages` simply uses `python.buildEnv` under the hood, it also
1128supports the `env` attribute. The `shell.nix` file from the previous section can
1129thus be also written like this:
1130
1131```nix
1132with import <nixpkgs> {};
1133
1134(python38.withPackages (ps: [ps.numpy ps.requests])).env
1135```
1136
1137In contrast to `python.buildEnv`, `python.withPackages` does not support the
1138more advanced options such as `ignoreCollisions = true` or `postBuild`. If you
1139need them, you have to use `python.buildEnv`.
1140
1141Python 2 namespace packages may provide `__init__.py` that collide. In that case
1142`python.buildEnv` should be used with `ignoreCollisions = true`.
1143
1144#### Setup hooks
1145
1146The following are setup hooks specifically for Python packages. Most of these
1147are used in `buildPythonPackage`.
1148
1149- `eggUnpackhook` to move an egg to the correct folder so it can be installed
1150 with the `eggInstallHook`
1151- `eggBuildHook` to skip building for eggs.
1152- `eggInstallHook` to install eggs.
1153- `flitBuildHook` to build a wheel using `flit`.
1154- `pipBuildHook` to build a wheel using `pip` and PEP 517. Note a build system
1155 (e.g. `setuptools` or `flit`) should still be added as `nativeBuildInput`.
1156- `pipInstallHook` to install wheels.
1157- `pytestCheckHook` to run tests with `pytest`. See [example usage](#using-pytestcheckhook).
1158- `pythonCatchConflictsHook` to check whether a Python package is not already existing.
1159- `pythonImportsCheckHook` to check whether importing the listed modules works.
1160- `pythonRemoveBinBytecode` to remove bytecode from the `/bin` folder.
1161- `setuptoolsBuildHook` to build a wheel using `setuptools`.
1162- `setuptoolsCheckHook` to run tests with `python setup.py test`.
1163- `venvShellHook` to source a Python 3 `venv` at the `venvDir` location. A
1164 `venv` is created if it does not yet exist. `postVenvCreation` can be used to
1165 to run commands only after venv is first created.
1166- `wheelUnpackHook` to move a wheel to the correct folder so it can be installed
1167 with the `pipInstallHook`.
1168
1169### Development mode
1170
1171Development or editable mode is supported. To develop Python packages
1172`buildPythonPackage` has additional logic inside `shellPhase` to run `pip
1173install -e . --prefix $TMPDIR/`for the package.
1174
1175Warning: `shellPhase` is executed only if `setup.py` exists.
1176
1177Given a `default.nix`:
1178```nix
1179with import <nixpkgs> {};
1180
1181pythonPackages.buildPythonPackage {
1182 name = "myproject";
1183 buildInputs = with pythonPackages; [ pyramid ];
1184
1185 src = ./.;
1186}
1187```
1188
1189Running `nix-shell` with no arguments should give you the environment in which
1190the package would be built with `nix-build`.
1191
1192Shortcut to setup environments with C headers/libraries and Python packages:
1193
1194```shell
1195nix-shell -p pythonPackages.pyramid zlib libjpeg git
1196```
1197
1198Note: There is a boolean value `lib.inNixShell` set to `true` if nix-shell is invoked.
1199
1200### Tools
1201
1202Packages inside nixpkgs are written by hand. However many tools exist in
1203community to help save time. No tool is preferred at the moment.
1204
1205- [pypi2nix](https://github.com/nix-community/pypi2nix): Generate Nix
1206 expressions for your Python project. Note that [sharing derivations from
1207 pypi2nix with nixpkgs is possible but not
1208 encouraged](https://github.com/nix-community/pypi2nix/issues/222#issuecomment-443497376).
1209- [nixpkgs-pytools](https://github.com/nix-community/nixpkgs-pytools)
1210- [poetry2nix](https://github.com/nix-community/poetry2nix)
1211
1212### Deterministic builds
1213
1214The Python interpreters are now built deterministically. Minor modifications had
1215to be made to the interpreters in order to generate deterministic bytecode. This
1216has security implications and is relevant for those using Python in a
1217`nix-shell`.
1218
1219When the environment variable `DETERMINISTIC_BUILD` is set, all bytecode will
1220have timestamp 1. The `buildPythonPackage` function sets `DETERMINISTIC_BUILD=1`
1221and [PYTHONHASHSEED=0](https://docs.python.org/3.8/using/cmdline.html#envvar-PYTHONHASHSEED).
1222Both are also exported in `nix-shell`.
1223
1224### Automatic tests
1225
1226It is recommended to test packages as part of the build process.
1227Source distributions (`sdist`) often include test files, but not always.
1228
1229By default the command `python setup.py test` is run as part of the
1230`checkPhase`, but often it is necessary to pass a custom `checkPhase`. An
1231example of such a situation is when `py.test` is used.
1232
1233#### Common issues
1234
1235* Non-working tests can often be deselected. By default `buildPythonPackage`
1236 runs `python setup.py test`. Most Python modules follows the standard test
1237 protocol where the pytest runner can be used instead. `py.test` supports a
1238 `-k` parameter to ignore test methods or classes:
1239
1240 ```nix
1241 buildPythonPackage {
1242 # ...
1243 # assumes the tests are located in tests
1244 checkInputs = [ pytest ];
1245 checkPhase = ''
1246 py.test -k 'not function_name and not other_function' tests
1247 '';
1248 }
1249 ```
1250* Tests that attempt to access `$HOME` can be fixed by using the following
1251 work-around before running tests (e.g. `preCheck`): `export HOME=$(mktemp -d)`
1252
1253## FAQ
1254
1255### How to solve circular dependencies?
1256
1257Consider the packages `A` and `B` that depend on each other. When packaging `B`,
1258a solution is to override package `A` not to depend on `B` as an input. The same
1259should also be done when packaging `A`.
1260
1261### How to override a Python package?
1262
1263We can override the interpreter and pass `packageOverrides`. In the following
1264example we rename the `pandas` package and build it.
1265
1266```nix
1267with import <nixpkgs> {};
1268
1269(let
1270 python = let
1271 packageOverrides = self: super: {
1272 pandas = super.pandas.overridePythonAttrs(old: {name="foo";});
1273 };
1274 in pkgs.python38.override {inherit packageOverrides;};
1275
1276in python.withPackages(ps: [ps.pandas])).env
1277```
1278
1279Using `nix-build` on this expression will build an environment that contains the
1280package `pandas` but with the new name `foo`.
1281
1282All packages in the package set will use the renamed package. A typical use case
1283is to switch to another version of a certain package. For example, in the
1284Nixpkgs repository we have multiple versions of `django` and `scipy`. In the
1285following example we use a different version of `scipy` and create an
1286environment that uses it. All packages in the Python package set will now use
1287the updated `scipy` version.
1288
1289```nix
1290with import <nixpkgs> {};
1291
1292( let
1293 packageOverrides = self: super: {
1294 scipy = super.scipy_0_17;
1295 };
1296 in (pkgs.python38.override {inherit packageOverrides;}).withPackages (ps: [ps.blaze])
1297).env
1298```
1299
1300The requested package `blaze` depends on `pandas` which itself depends on `scipy`.
1301
1302If you want the whole of Nixpkgs to use your modifications, then you can use
1303`overlays` as explained in this manual. In the following example we build a
1304`inkscape` using a different version of `numpy`.
1305
1306```nix
1307let
1308 pkgs = import <nixpkgs> {};
1309 newpkgs = import pkgs.path { overlays = [ (self: super: {
1310 python38 = let
1311 packageOverrides = python-self: python-super: {
1312 numpy = python-super.numpy_1_18;
1313 };
1314 in super.python38.override {inherit packageOverrides;};
1315 } ) ]; };
1316in newpkgs.inkscape
1317```
1318
1319### `python setup.py bdist_wheel` cannot create .whl
1320
1321Executing `python setup.py bdist_wheel` in a `nix-shell `fails with
1322```
1323ValueError: ZIP does not support timestamps before 1980
1324```
1325
1326This is because files from the Nix store (which have a timestamp of the UNIX
1327epoch of January 1, 1970) are included in the .ZIP, but .ZIP archives follow the
1328DOS convention of counting timestamps from 1980.
1329
1330The command `bdist_wheel` reads the `SOURCE_DATE_EPOCH` environment variable,
1331which `nix-shell` sets to 1. Unsetting this variable or giving it a value
1332corresponding to 1980 or later enables building wheels.
1333
1334Use 1980 as timestamp:
1335
1336```shell
1337nix-shell --run "SOURCE_DATE_EPOCH=315532800 python3 setup.py bdist_wheel"
1338```
1339
1340or the current time:
1341
1342```shell
1343nix-shell --run "SOURCE_DATE_EPOCH=$(date +%s) python3 setup.py bdist_wheel"
1344```
1345
1346or unset `SOURCE_DATE_EPOCH`:
1347
1348```shell
1349nix-shell --run "unset SOURCE_DATE_EPOCH; python3 setup.py bdist_wheel"
1350```
1351
1352### `install_data` / `data_files` problems
1353
1354If you get the following error:
1355
1356```
1357could not create '/nix/store/6l1bvljpy8gazlsw2aw9skwwp4pmvyxw-python-2.7.8/etc':
1358Permission denied
1359```
1360
1361This is a [known bug](https://github.com/pypa/setuptools/issues/130) in
1362`setuptools`. Setuptools `install_data` does not respect `--prefix`. An example
1363of such package using the feature is `pkgs/tools/X11/xpra/default.nix`.
1364
1365As workaround install it as an extra `preInstall` step:
1366
1367```shell
1368${python.interpreter} setup.py install_data --install-dir=$out --root=$out
1369sed -i '/ = data\_files/d' setup.py
1370```
1371
1372### Rationale of non-existent global site-packages
1373
1374On most operating systems a global `site-packages` is maintained. This however
1375becomes problematic if you want to run multiple Python versions or have multiple
1376versions of certain libraries for your projects. Generally, you would solve such
1377issues by creating virtual environments using `virtualenv`.
1378
1379On Nix each package has an isolated dependency tree which, in the case of
1380Python, guarantees the right versions of the interpreter and libraries or
1381packages are available. There is therefore no need to maintain a global `site-packages`.
1382
1383If you want to create a Python environment for development, then the recommended
1384method is to use `nix-shell`, either with or without the `python.buildEnv`
1385function.
1386
1387### How to consume Python modules using pip in a virtual environment like I am used to on other Operating Systems?
1388
1389While this approach is not very idiomatic from Nix perspective, it can still be
1390useful when dealing with pre-existing projects or in situations where it's not
1391feasible or desired to write derivations for all required dependencies.
1392
1393This is an example of a `default.nix` for a `nix-shell`, which allows to consume
1394a virtual environment created by `venv`, and install Python modules through
1395`pip` the traditional way.
1396
1397Create this `default.nix` file, together with a `requirements.txt` and simply
1398execute `nix-shell`.
1399
1400```nix
1401with import <nixpkgs> { };
1402
1403let
1404 pythonPackages = python3Packages;
1405in pkgs.mkShell rec {
1406 name = "impurePythonEnv";
1407 venvDir = "./.venv";
1408 buildInputs = [
1409 # A Python interpreter including the 'venv' module is required to bootstrap
1410 # the environment.
1411 pythonPackages.python
1412
1413 # This execute some shell code to initialize a venv in $venvDir before
1414 # dropping into the shell
1415 pythonPackages.venvShellHook
1416
1417 # Those are dependencies that we would like to use from nixpkgs, which will
1418 # add them to PYTHONPATH and thus make them accessible from within the venv.
1419 pythonPackages.numpy
1420 pythonPackages.requests
1421
1422 # In this particular example, in order to compile any binary extensions they may
1423 # require, the Python modules listed in the hypothetical requirements.txt need
1424 # the following packages to be installed locally:
1425 taglib
1426 openssl
1427 git
1428 libxml2
1429 libxslt
1430 libzip
1431 zlib
1432 ];
1433
1434 # Run this command, only after creating the virtual environment
1435 postVenvCreation = ''
1436 unset SOURCE_DATE_EPOCH
1437 pip install -r requirements.txt
1438 '';
1439
1440 # Now we can execute any commands within the virtual environment.
1441 # This is optional and can be left out to run pip manually.
1442 postShellHook = ''
1443 # allow pip to install wheels
1444 unset SOURCE_DATE_EPOCH
1445 '';
1446
1447}
1448```
1449
1450In case the supplied venvShellHook is insufficient, or when Python 2 support is
1451needed, you can define your own shell hook and adapt to your needs like in the
1452following example:
1453
1454```nix
1455with import <nixpkgs> { };
1456
1457let
1458 venvDir = "./.venv";
1459 pythonPackages = python3Packages;
1460in pkgs.mkShell rec {
1461 name = "impurePythonEnv";
1462 buildInputs = [
1463 pythonPackages.python
1464 # Needed when using python 2.7
1465 # pythonPackages.virtualenv
1466 # ...
1467 ];
1468
1469 # This is very close to how venvShellHook is implemented, but
1470 # adapted to use 'virtualenv'
1471 shellHook = ''
1472 SOURCE_DATE_EPOCH=$(date +%s)
1473
1474 if [ -d "${venvDir}" ]; then
1475 echo "Skipping venv creation, '${venvDir}' already exists"
1476 else
1477 echo "Creating new venv environment in path: '${venvDir}'"
1478 # Note that the module venv was only introduced in python 3, so for 2.7
1479 # this needs to be replaced with a call to virtualenv
1480 ${pythonPackages.python.interpreter} -m venv "${venvDir}"
1481 fi
1482
1483 # Under some circumstances it might be necessary to add your virtual
1484 # environment to PYTHONPATH, which you can do here too;
1485 # PYTHONPATH=$PWD/${venvDir}/${pythonPackages.python.sitePackages}/:$PYTHONPATH
1486
1487 source "${venvDir}/bin/activate"
1488
1489 # As in the previous example, this is optional.
1490 pip install -r requirements.txt
1491 '';
1492}
1493```
1494
1495Note that the `pip install` is an imperative action. So every time `nix-shell`
1496is executed it will attempt to download the Python modules listed in
1497requirements.txt. However these will be cached locally within the `virtualenv`
1498folder and not downloaded again.
1499
1500### How to override a Python package from `configuration.nix`?
1501
1502If you need to change a package's attribute(s) from `configuration.nix` you could do:
1503
1504```nix
1505 nixpkgs.config.packageOverrides = super: {
1506 python = super.python.override {
1507 packageOverrides = python-self: python-super: {
1508 twisted = python-super.twisted.overrideAttrs (oldAttrs: {
1509 src = super.fetchPipy {
1510 pname = "twisted";
1511 version = "19.10.0";
1512 sha256 = "7394ba7f272ae722a74f3d969dcf599bc4ef093bc392038748a490f1724a515d";
1513 extension = "tar.bz2";
1514 };
1515 });
1516 };
1517 };
1518 };
1519```
1520
1521`pythonPackages.twisted` is now globally overridden.
1522All packages and also all NixOS services that reference `twisted`
1523(such as `services.buildbot-worker`) now use the new definition.
1524Note that `python-super` refers to the old package set and `python-self`
1525to the new, overridden version.
1526
1527To modify only a Python package set instead of a whole Python derivation, use
1528this snippet:
1529
1530```nix
1531 myPythonPackages = pythonPackages.override {
1532 overrides = self: super: {
1533 twisted = ...;
1534 };
1535 }
1536```
1537
1538### How to override a Python package using overlays?
1539
1540Use the following overlay template:
1541
1542```nix
1543self: super: {
1544 python = super.python.override {
1545 packageOverrides = python-self: python-super: {
1546 twisted = python-super.twisted.overrideAttrs (oldAttrs: {
1547 src = super.fetchPypi {
1548 pname = "twisted";
1549 version = "19.10.0";
1550 sha256 = "7394ba7f272ae722a74f3d969dcf599bc4ef093bc392038748a490f1724a515d";
1551 extension = "tar.bz2";
1552 };
1553 });
1554 };
1555 };
1556}
1557```
1558
1559### How to use Intel's MKL with numpy and scipy?
1560
1561MKL can be configured using an overlay. See the section "[Using overlays to
1562configure alternatives](#sec-overlays-alternatives-blas-lapack)".
1563
1564### What inputs do `setup_requires`, `install_requires` and `tests_require` map to?
1565
1566In a `setup.py` or `setup.cfg` it is common to declare dependencies:
1567
1568* `setup_requires` corresponds to `nativeBuildInputs`
1569* `install_requires` corresponds to `propagatedBuildInputs`
1570* `tests_require` corresponds to `checkInputs`
1571
1572## Contributing
1573
1574### Contributing guidelines
1575
1576The following rules are desired to be respected:
1577
1578* Python libraries are called from `python-packages.nix` and packaged with
1579 `buildPythonPackage`. The expression of a library should be in
1580 `pkgs/development/python-modules/<name>/default.nix`.
1581* Python applications live outside of `python-packages.nix` and are packaged
1582 with `buildPythonApplication`.
1583* Make sure libraries build for all Python interpreters.
1584* By default we enable tests. Make sure the tests are found and, in the case of
1585 libraries, are passing for all interpreters. If certain tests fail they can be
1586 disabled individually. Try to avoid disabling the tests altogether. In any
1587 case, when you disable tests, leave a comment explaining why.
1588* Commit names of Python libraries should reflect that they are Python
1589 libraries, so write for example `pythonPackages.numpy: 1.11 -> 1.12`.
1590* Attribute names in `python-packages.nix` as well as `pname`s should match the
1591 library's name on PyPI, but be normalized according to [PEP
1592 0503](https://www.python.org/dev/peps/pep-0503/#normalized-names). This means
1593 that characters should be converted to lowercase and `.` and `_` should be
1594 replaced by a single `-` (foo-bar-baz instead of Foo__Bar.baz).
1595 If necessary, `pname` has to be given a different value within `fetchPypi`.
1596* Attribute names in `python-packages.nix` should be sorted alphanumerically to
1597 avoid merge conflicts and ease locating attributes.