pyrox.dev
1# Python
2
3## User Guide
4
5### Using Python
6
7#### Overview
8
9Several versions of the Python interpreter are available on Nix, as well as a
10high amount of packages. The attribute `python` refers to the default
11interpreter, which is currently CPython 2.7. It is also possible to refer to
12specific versions, e.g. `python35` refers to CPython 3.5, and `pypy` refers to
13the default PyPy interpreter.
14
15Python is used a lot, and in different ways. This affects also how it is
16packaged. In the case of Python on Nix, an important distinction is made between
17whether the package is considered primarily an application, or whether it should
18be used as a library, i.e., of primary interest are the modules in
19`site-packages` that should be importable.
20
21In the Nixpkgs tree Python applications can be found throughout, depending on
22what they do, and are called from the main package set. Python libraries,
23however, are in separate sets, with one set per interpreter version.
24
25The interpreters have several common attributes. One of these attributes is
26`pkgs`, which is a package set of Python libraries for this specific
27interpreter. E.g., the `toolz` package corresponding to the default interpreter
28is `python.pkgs.toolz`, and the CPython 3.5 version is `python35.pkgs.toolz`.
29The main package set contains aliases to these package sets, e.g.
30`pythonPackages` refers to `python.pkgs` and `python35Packages` to
31`python35.pkgs`.
32
33#### Installing Python and packages
34
35The Nix and NixOS manuals explain how packages are generally installed. In the
36case of Python and Nix, it is important to make a distinction between whether the
37package is considered an application or a library.
38
39Applications on Nix are typically installed into your user
40profile imperatively using `nix-env -i`, and on NixOS declaratively by adding the
41package name to `environment.systemPackages` in `/etc/nixos/configuration.nix`.
42Dependencies such as libraries are automatically installed and should not be
43installed explicitly.
44
45The same goes for Python applications and libraries. Python applications can be
46installed in your profile. But Python libraries you would like to use for
47development cannot be installed, at least not individually, because they won't
48be able to find each other resulting in import errors. Instead, it is possible
49to create an environment with `python.buildEnv` or `python.withPackages` where
50the interpreter and other executables are able to find each other and all of the
51modules.
52
53In the following examples we create an environment with Python 3.5, `numpy` and
54`toolz`. As you may imagine, there is one limitation here, and that's that
55you can install only one environment at a time. You will notice the complaints
56about collisions when you try to install a second environment.
57
58##### Environment defined in separate `.nix` file
59
60Create a file, e.g. `build.nix`, with the following expression
61```nix
62with import <nixpkgs> {};
63
64python35.withPackages (ps: with ps; [ numpy toolz ])
65```
66and install it in your profile with
67```shell
68nix-env -if build.nix
69```
70Now you can use the Python interpreter, as well as the extra packages (`numpy`,
71`toolz`) that you added to the environment.
72
73##### Environment defined in `~/.config/nixpkgs/config.nix`
74
75If you prefer to, you could also add the environment as a package override to the Nixpkgs set, e.g.
76using `config.nix`,
77```nix
78{ # ...
79
80 packageOverrides = pkgs: with pkgs; {
81 myEnv = python35.withPackages (ps: with ps; [ numpy toolz ]);
82 };
83}
84```
85and install it in your profile with
86```shell
87nix-env -iA nixpkgs.myEnv
88```
89The environment is is installed by referring to the attribute, and considering
90the `nixpkgs` channel was used.
91
92##### Environment defined in `/etc/nixos/configuration.nix`
93
94For the sake of completeness, here's another example how to install the environment system-wide.
95
96```nix
97{ # ...
98
99 environment.systemPackages = with pkgs; [
100 (python35.withPackages(ps: with ps; [ numpy toolz ]))
101 ];
102}
103```
104
105#### Temporary Python environment with `nix-shell`
106
107The examples in the previous section showed how to install a Python environment
108into a profile. For development you may need to use multiple environments.
109`nix-shell` gives the possibility to temporarily load another environment, akin
110to `virtualenv`.
111
112There are two methods for loading a shell with Python packages. The first and recommended method
113is to create an environment with `python.buildEnv` or `python.withPackages` and load that. E.g.
114```sh
115$ nix-shell -p 'python35.withPackages(ps: with ps; [ numpy toolz ])'
116```
117opens a shell from which you can launch the interpreter
118```sh
119[nix-shell:~] python3
120```
121The other method, which is not recommended, does not create an environment and requires you to list the packages directly,
122
123```sh
124$ nix-shell -p python35.pkgs.numpy python35.pkgs.toolz
125```
126Again, it is possible to launch the interpreter from the shell.
127The Python interpreter has the attribute `pkgs` which contains all Python libraries for that specific interpreter.
128
129##### Load environment from `.nix` expression
130As explained in the Nix manual, `nix-shell` can also load an
131expression from a `.nix` file. Say we want to have Python 3.5, `numpy`
132and `toolz`, like before, in an environment. Consider a `shell.nix` file
133with
134```nix
135with import <nixpkgs> {};
136
137(python35.withPackages (ps: [ps.numpy ps.toolz])).env
138```
139Executing `nix-shell` gives you again a Nix shell from which you can run Python.
140
141What's happening here?
142
1431. We begin with importing the Nix Packages collections. `import <nixpkgs>` imports the `<nixpkgs>` function, `{}` calls it and the `with` statement brings all attributes of `nixpkgs` in the local scope. These attributes form the main package set.
1442. Then we create a Python 3.5 environment with the `withPackages` function.
1453. The `withPackages` function expects us to provide a function as an argument that takes the set of all python packages and returns a list of packages to include in the environment. Here, we select the packages `numpy` and `toolz` from the package set.
146
147##### Execute command with `--run`
148A convenient option with `nix-shell` is the `--run`
149option, with which you can execute a command in the `nix-shell`. We can
150e.g. directly open a Python shell
151```sh
152$ nix-shell -p python35Packages.numpy python35Packages.toolz --run "python3"
153```
154or run a script
155```sh
156$ nix-shell -p python35Packages.numpy python35Packages.toolz --run "python3 myscript.py"
157```
158
159##### `nix-shell` as shebang
160In fact, for the second use case, there is a more convenient method. You can
161add a [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) to your script
162specifying which dependencies `nix-shell` needs. With the following shebang, you
163can just execute `./myscript.py`, and it will make available all dependencies and
164run the script in the `python3` shell.
165
166```py
167#! /usr/bin/env nix-shell
168#! nix-shell -i python3 -p "python3.withPackages(ps: [ps.numpy])"
169
170import numpy
171
172print(numpy.__version__)
173```
174
175### Developing with Python
176
177Now that you know how to get a working Python environment with Nix, it is time
178to go forward and start actually developing with Python. We will first have a
179look at how Python packages are packaged on Nix. Then, we will look at how you
180can use development mode with your code.
181
182#### Packaging a library
183
184With Nix all packages are built by functions. The main function in Nix for
185building Python libraries is `buildPythonPackage`. Let's see how we can build the
186`toolz` package.
187
188```nix
189{ # ...
190
191 toolz = buildPythonPackage rec {
192 pname = "toolz";
193 version = "0.7.4";
194
195 src = fetchPypi {
196 inherit pname version;
197 sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed05afd";
198 };
199
200 doCheck = false;
201
202 meta = {
203 homepage = "https://github.com/pytoolz/toolz/";
204 description = "List processing tools and functional utilities";
205 license = licenses.bsd3;
206 maintainers = with maintainers; [ fridh ];
207 };
208 };
209}
210```
211
212What happens here? The function `buildPythonPackage` is called and as argument
213it accepts a set. In this case the set is a recursive set, `rec`. One of the
214arguments is the name of the package, which consists of a basename (generally
215following the name on PyPi) and a version. Another argument, `src` specifies the
216source, which in this case is fetched from PyPI using the helper function
217`fetchPypi`. The argument `doCheck` is used to set whether tests should be run
218when building the package. Furthermore, we specify some (optional) meta
219information. The output of the function is a derivation.
220
221An expression for `toolz` can be found in the Nixpkgs repository. As explained
222in the introduction of this Python section, a derivation of `toolz` is available
223for each interpreter version, e.g. `python35.pkgs.toolz` refers to the `toolz`
224derivation corresponding to the CPython 3.5 interpreter.
225The above example works when you're directly working on
226`pkgs/top-level/python-packages.nix` in the Nixpkgs repository. Often though,
227you will want to test a Nix expression outside of the Nixpkgs tree.
228
229The following expression creates a derivation for the `toolz` package,
230and adds it along with a `numpy` package to a Python environment.
231
232```nix
233with import <nixpkgs> {};
234
235( let
236 my_toolz = python35.pkgs.buildPythonPackage rec {
237 pname = "toolz";
238 version = "0.7.4";
239
240 src = python35.pkgs.fetchPypi {
241 inherit pname version;
242 sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed05afd";
243 };
244
245 doCheck = false;
246
247 meta = {
248 homepage = "https://github.com/pytoolz/toolz/";
249 description = "List processing tools and functional utilities";
250 };
251 };
252
253 in python35.withPackages (ps: [ps.numpy my_toolz])
254).env
255```
256Executing `nix-shell` will result in an environment in which you can use
257Python 3.5 and the `toolz` package. As you can see we had to explicitly mention
258for which Python version we want to build a package.
259
260So, what did we do here? Well, we took the Nix expression that we used earlier
261to build a Python environment, and said that we wanted to include our own
262version of `toolz`, named `my_toolz`. To introduce our own package in the scope
263of `withPackages` we used a `let` expression. You can see that we used
264`ps.numpy` to select numpy from the nixpkgs package set (`ps`). We did not take
265`toolz` from the Nixpkgs package set this time, but instead took our own version
266that we introduced with the `let` expression.
267
268#### Handling dependencies
269
270Our example, `toolz`, does not have any dependencies on other Python
271packages or system libraries. According to the manual, `buildPythonPackage`
272uses the arguments `buildInputs` and `propagatedBuildInputs` to specify dependencies. If something is
273exclusively a build-time dependency, then the dependency should be included as a
274`buildInput`, but if it is (also) a runtime dependency, then it should be added
275to `propagatedBuildInputs`. Test dependencies are considered build-time dependencies.
276
277The following example shows which arguments are given to `buildPythonPackage` in
278order to build [`datashape`](https://github.com/blaze/datashape).
279
280```nix
281{ # ...
282
283 datashape = buildPythonPackage rec {
284 pname = "datashape";
285 version = "0.4.7";
286
287 src = fetchPypi {
288 inherit pname version;
289 sha256 = "14b2ef766d4c9652ab813182e866f493475e65e558bed0822e38bf07bba1a278";
290 };
291
292 checkInputs = with self; [ pytest ];
293 propagatedBuildInputs = with self; [ numpy multipledispatch dateutil ];
294
295 meta = {
296 homepage = https://github.com/ContinuumIO/datashape;
297 description = "A data description language";
298 license = licenses.bsd2;
299 maintainers = with maintainers; [ fridh ];
300 };
301 };
302}
303```
304
305We can see several runtime dependencies, `numpy`, `multipledispatch`, and
306`dateutil`. Furthermore, we have one `buildInput`, i.e. `pytest`. `pytest` is a
307test runner and is only used during the `checkPhase` and is therefore not added
308to `propagatedBuildInputs`.
309
310In the previous case we had only dependencies on other Python packages to consider.
311Occasionally you have also system libraries to consider. E.g., `lxml` provides
312Python bindings to `libxml2` and `libxslt`. These libraries are only required
313when building the bindings and are therefore added as `buildInputs`.
314
315```nix
316{ # ...
317
318 lxml = buildPythonPackage rec {
319 pname = "lxml";
320 version = "3.4.4";
321
322 src = fetchPypi {
323 inherit pname version;
324 sha256 = "16a0fa97hym9ysdk3rmqz32xdjqmy4w34ld3rm3jf5viqjx65lxk";
325 };
326
327 buildInputs = with self; [ pkgs.libxml2 pkgs.libxslt ];
328
329 meta = {
330 description = "Pythonic binding for the libxml2 and libxslt libraries";
331 homepage = https://lxml.de;
332 license = licenses.bsd3;
333 maintainers = with maintainers; [ sjourdois ];
334 };
335 };
336}
337```
338
339In this example `lxml` and Nix are able to work out exactly where the relevant
340files of the dependencies are. This is not always the case.
341
342The example below shows bindings to The Fastest Fourier Transform in the West, commonly known as
343FFTW. On Nix we have separate packages of FFTW for the different types of floats
344(`"single"`, `"double"`, `"long-double"`). The bindings need all three types,
345and therefore we add all three as `buildInputs`. The bindings don't expect to
346find each of them in a different folder, and therefore we have to set `LDFLAGS`
347and `CFLAGS`.
348
349```nix
350{ # ...
351
352 pyfftw = buildPythonPackage rec {
353 pname = "pyFFTW";
354 version = "0.9.2";
355
356 src = fetchPypi {
357 inherit pname version;
358 sha256 = "f6bbb6afa93085409ab24885a1a3cdb8909f095a142f4d49e346f2bd1b789074";
359 };
360
361 buildInputs = [ pkgs.fftw pkgs.fftwFloat pkgs.fftwLongDouble];
362
363 propagatedBuildInputs = with self; [ numpy scipy ];
364
365 # Tests cannot import pyfftw. pyfftw works fine though.
366 doCheck = false;
367
368 preConfigure = ''
369 export LDFLAGS="-L${pkgs.fftw.dev}/lib -L${pkgs.fftwFloat.out}/lib -L${pkgs.fftwLongDouble.out}/lib"
370 export CFLAGS="-I${pkgs.fftw.dev}/include -I${pkgs.fftwFloat.dev}/include -I${pkgs.fftwLongDouble.dev}/include"
371 '';
372
373 meta = {
374 description = "A pythonic wrapper around FFTW, the FFT library, presenting a unified interface for all the supported transforms";
375 homepage = http://hgomersall.github.com/pyFFTW/;
376 license = with licenses; [ bsd2 bsd3 ];
377 maintainers = with maintainers; [ fridh ];
378 };
379 };
380}
381```
382Note also the line `doCheck = false;`, we explicitly disabled running the test-suite.
383
384
385#### Develop local package
386
387As a Python developer you're likely aware of [development mode](http://setuptools.readthedocs.io/en/latest/setuptools.html#development-mode) (`python setup.py develop`);
388instead of installing the package this command creates a special link to the project code.
389That way, you can run updated code without having to reinstall after each and every change you make.
390Development mode is also available. Let's see how you can use it.
391
392In the previous Nix expression the source was fetched from an url. We can also refer to a local source instead using
393`src = ./path/to/source/tree;`
394
395If we create a `shell.nix` file which calls `buildPythonPackage`, and if `src`
396is a local source, and if the local source has a `setup.py`, then development
397mode is activated.
398
399In the following example we create a simple environment that
400has a Python 3.5 version of our package in it, as well as its dependencies and
401other packages we like to have in the environment, all specified with `propagatedBuildInputs`.
402Indeed, we can just add any package we like to have in our environment to `propagatedBuildInputs`.
403
404```nix
405with import <nixpkgs> {};
406with pkgs.python35Packages;
407
408buildPythonPackage rec {
409 name = "mypackage";
410 src = ./path/to/package/source;
411 propagatedBuildInputs = [ pytest numpy pkgs.libsndfile ];
412}
413```
414
415It is important to note that due to how development mode is implemented on Nix it is not possible to have multiple packages simultaneously in development mode.
416
417
418### Organising your packages
419
420So far we discussed how you can use Python on Nix, and how you can develop with
421it. We've looked at how you write expressions to package Python packages, and we
422looked at how you can create environments in which specified packages are
423available.
424
425At some point you'll likely have multiple packages which you would
426like to be able to use in different projects. In order to minimise unnecessary
427duplication we now look at how you can maintain a repository with your
428own packages. The important functions here are `import` and `callPackage`.
429
430### Including a derivation using `callPackage`
431
432Earlier we created a Python environment using `withPackages`, and included the
433`toolz` package via a `let` expression.
434Let's split the package definition from the environment definition.
435
436We first create a function that builds `toolz` in `~/path/to/toolz/release.nix`
437
438```nix
439{ lib, pkgs, buildPythonPackage }:
440
441buildPythonPackage rec {
442 pname = "toolz";
443 version = "0.7.4";
444
445 src = fetchPypi {
446 inherit pname version;
447 sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed05afd";
448 };
449
450 meta = with lib; {
451 homepage = "http://github.com/pytoolz/toolz/";
452 description = "List processing tools and functional utilities";
453 license = licenses.bsd3;
454 maintainers = with maintainers; [ fridh ];
455 };
456}
457```
458
459It takes two arguments, `pkgs` and `buildPythonPackage`.
460We now call this function using `callPackage` in the definition of our environment
461
462```nix
463with import <nixpkgs> {};
464
465( let
466 toolz = pkgs.callPackage /path/to/toolz/release.nix {
467 pkgs = pkgs;
468 buildPythonPackage = pkgs.python35Packages.buildPythonPackage;
469 };
470 in pkgs.python35.withPackages (ps: [ ps.numpy toolz ])
471).env
472```
473
474Important to remember is that the Python version for which the package is made
475depends on the `python` derivation that is passed to `buildPythonPackage`. Nix
476tries to automatically pass arguments when possible, which is why generally you
477don't explicitly define which `python` derivation should be used. In the above
478example we use `buildPythonPackage` that is part of the set `python35Packages`,
479and in this case the `python35` interpreter is automatically used.
480
481
482
483## Reference
484
485### Interpreters
486
487Versions 2.7, 3.4, 3.5, 3.6 and 3.7 of the CPython interpreter are available as
488respectively `python27`, `python34`, `python35` and `python36`. The PyPy interpreter
489is available as `pypy`. The aliases `python2` and `python3` correspond to respectively `python27` and
490`python35`. The default interpreter, `python`, maps to `python2`.
491The Nix expressions for the interpreters can be found in
492`pkgs/development/interpreters/python`.
493
494All packages depending on any Python interpreter get appended
495`out/{python.sitePackages}` to `$PYTHONPATH` if such directory
496exists.
497
498#### Missing `tkinter` module standard library
499
500To reduce closure size the `Tkinter`/`tkinter` is available as a separate package, `pythonPackages.tkinter`.
501
502#### Attributes on interpreters packages
503
504Each interpreter has the following attributes:
505
506- `libPrefix`. Name of the folder in `${python}/lib/` for corresponding interpreter.
507- `interpreter`. Alias for `${python}/bin/${executable}`.
508- `buildEnv`. Function to build python interpreter environments with extra packages bundled together. See section *python.buildEnv function* for usage and documentation.
509- `withPackages`. Simpler interface to `buildEnv`. See section *python.withPackages function* for usage and documentation.
510- `sitePackages`. Alias for `lib/${libPrefix}/site-packages`.
511- `executable`. Name of the interpreter executable, e.g. `python3.4`.
512- `pkgs`. Set of Python packages for that specific interpreter. The package set can be modified by overriding the interpreter and passing `packageOverrides`.
513
514### Building packages and applications
515
516Python libraries and applications that use `setuptools` or
517`distutils` are typically build with respectively the `buildPythonPackage` and
518`buildPythonApplication` functions. These two functions also support installing a `wheel`.
519
520All Python packages reside in `pkgs/top-level/python-packages.nix` and all
521applications elsewhere. In case a package is used as both a library and an application,
522then the package should be in `pkgs/top-level/python-packages.nix` since only those packages are made
523available for all interpreter versions. The preferred location for library expressions is in
524`pkgs/development/python-modules`. It is important that these packages are
525called from `pkgs/top-level/python-packages.nix` and not elsewhere, to guarantee
526the right version of the package is built.
527
528Based on the packages defined in `pkgs/top-level/python-packages.nix` an
529attribute set is created for each available Python interpreter. The available
530sets are
531
532* `pkgs.python27Packages`
533* `pkgs.python34Packages`
534* `pkgs.python35Packages`
535* `pkgs.python36Packages`
536* `pkgs.python37Packages`
537* `pkgs.pypyPackages`
538
539and the aliases
540
541* `pkgs.python2Packages` pointing to `pkgs.python27Packages`
542* `pkgs.python3Packages` pointing to `pkgs.python36Packages`
543* `pkgs.pythonPackages` pointing to `pkgs.python2Packages`
544
545#### `buildPythonPackage` function
546
547The `buildPythonPackage` function is implemented in
548`pkgs/development/interpreters/python/build-python-package.nix`
549
550The following is an example:
551```nix
552
553buildPythonPackage rec {
554 version = "3.3.1";
555 pname = "pytest";
556
557 preCheck = ''
558 # don't test bash builtins
559 rm testing/test_argcomplete.py
560 '';
561
562 src = fetchPypi {
563 inherit pname version;
564 sha256 = "cf8436dc59d8695346fcd3ab296de46425ecab00d64096cebe79fb51ecb2eb93";
565 };
566
567 checkInputs = [ hypothesis ];
568 buildInputs = [ setuptools_scm ];
569 propagatedBuildInputs = [ attrs py setuptools six pluggy ];
570
571 meta = with stdenv.lib; {
572 maintainers = with maintainers; [ domenkozar lovek323 madjar lsix ];
573 description = "Framework for writing tests";
574 };
575}
576
577```
578
579The `buildPythonPackage` mainly does four things:
580
581* In the `buildPhase`, it calls `${python.interpreter} setup.py bdist_wheel` to
582 build a wheel binary zipfile.
583* In the `installPhase`, it installs the wheel file using `pip install *.whl`.
584* In the `postFixup` phase, the `wrapPythonPrograms` bash function is called to
585 wrap all programs in the `$out/bin/*` directory to include `$PATH`
586 environment variable and add dependent libraries to script's `sys.path`.
587* In the `installCheck` phase, `${python.interpreter} setup.py test` is ran.
588
589As in Perl, dependencies on other Python packages can be specified in the
590`buildInputs` and `propagatedBuildInputs` attributes. If something is
591exclusively a build-time dependency, use `buildInputs`; if it is (also) a runtime
592dependency, use `propagatedBuildInputs`.
593
594By default tests are run because `doCheck = true`. Test dependencies, like
595e.g. the test runner, should be added to `checkInputs`.
596
597By default `meta.platforms` is set to the same value
598as the interpreter unless overridden otherwise.
599
600##### `buildPythonPackage` parameters
601
602All parameters from `stdenv.mkDerivation` function are still supported. The following are specific to `buildPythonPackage`:
603
604* `catchConflicts ? true`: If `true`, abort package build if a package name appears more than once in dependency tree. Default is `true`.
605* `checkInputs ? []`: Dependencies needed for running the `checkPhase`. These are added to `buildInputs` when `doCheck = true`.
606* `disabled` ? false: If `true`, package is not build for the particular Python interpreter version.
607* `dontWrapPythonPrograms ? false`: Skip wrapping of python programs.
608* `installFlags ? []`: A list of strings. Arguments to be passed to `pip install`. To pass options to `python setup.py install`, use `--install-option`. E.g., `installFlags=["--install-option='--cpp_implementation'"].
609* `format ? "setuptools"`: Format of the source. Valid options are `"setuptools"`, `"flit"`, `"wheel"`, and `"other"`. `"setuptools"` is for when the source has a `setup.py` and `setuptools` is used to build a wheel, `flit`, in case `flit` should be used to build a wheel, and `wheel` in case a wheel is provided. Use `other` when a custom `buildPhase` and/or `installPhase` is needed.
610* `makeWrapperArgs ? []`: A list of strings. Arguments to be passed to `makeWrapper`, which wraps generated binaries. By default, the arguments to `makeWrapper` set `PATH` and `PYTHONPATH` environment variables before calling the binary. Additional arguments here can allow a developer to set environment variables which will be available when the binary is run. For example, `makeWrapperArgs = ["--set FOO BAR" "--set BAZ QUX"]`.
611* `namePrefix`: Prepends text to `${name}` parameter. In case of libraries, this defaults to `"python3.5-"` for Python 3.5, etc., and in case of applications to `""`.
612* `pythonPath ? []`: List of packages to be added into `$PYTHONPATH`. Packages in `pythonPath` are not propagated (contrary to `propagatedBuildInputs`).
613* `preShellHook`: Hook to execute commands before `shellHook`.
614* `postShellHook`: Hook to execute commands after `shellHook`.
615* `removeBinByteCode ? true`: Remove bytecode from `/bin`. Bytecode is only created when the filenames end with `.py`.
616* `setupPyBuildFlags ? []`: List of flags passed to `setup.py build_ext` command.
617
618##### Overriding Python packages
619
620The `buildPythonPackage` function has a `overridePythonAttrs` method that
621can be used to override the package. In the following example we create an
622environment where we have the `blaze` package using an older version of `pandas`.
623We override first the Python interpreter and pass
624`packageOverrides` which contains the overrides for packages in the package set.
625
626```nix
627with import <nixpkgs> {};
628
629(let
630 python = let
631 packageOverrides = self: super: {
632 pandas = super.pandas.overridePythonAttrs(old: rec {
633 version = "0.19.1";
634 src = super.fetchPypi {
635 pname = "pandas";
636 inherit version;
637 sha256 = "08blshqj9zj1wyjhhw3kl2vas75vhhicvv72flvf1z3jvapgw295";
638 };
639 });
640 };
641 in pkgs.python3.override {inherit packageOverrides;};
642
643in python.withPackages(ps: [ps.blaze])).env
644```
645
646#### `buildPythonApplication` function
647
648The `buildPythonApplication` function is practically the same as
649`buildPythonPackage`. The main purpose of this function is to build a Python
650package where one is interested only in the executables, and not importable
651modules. For that reason, when adding this package to a `python.buildEnv`, the
652modules won't be made available.
653
654Another difference is that `buildPythonPackage` by default prefixes the names of
655the packages with the version of the interpreter. Because this is irrelevant for
656applications, the prefix is omitted.
657
658#### `toPythonApplication` function
659
660A distinction is made between applications and libraries, however, sometimes a
661package is used as both. In this case the package is added as a library to
662`python-packages.nix` and as an application to `all-packages.nix`. To reduce
663duplication the `toPythonApplication` can be used to convert a library to an
664application.
665
666The Nix expression shall use `buildPythonPackage` and be called from
667`python-packages.nix`. A reference shall be created from `all-packages.nix` to
668the attribute in `python-packages.nix`, and the `toPythonApplication` shall be
669applied to the reference:
670```nix
671youtube-dl = with pythonPackages; toPythonApplication youtube-dl;
672```
673
674#### `toPythonModule` function
675
676In some cases, such as bindings, a package is created using
677`stdenv.mkDerivation` and added as attribute in `all-packages.nix`.
678The Python bindings should be made available from `python-packages.nix`.
679The `toPythonModule` function takes a derivation and makes certain Python-specific modifications.
680```nix
681opencv = toPythonModule (pkgs.opencv.override {
682 enablePython = true;
683 pythonPackages = self;
684});
685```
686Do pay attention to passing in the right Python version!
687
688#### `python.buildEnv` function
689
690Python environments can be created using the low-level `pkgs.buildEnv` function.
691This example shows how to create an environment that has the Pyramid Web Framework.
692Saving the following as `default.nix`
693```nix
694with import <nixpkgs> {};
695
696python.buildEnv.override {
697 extraLibs = [ pkgs.pythonPackages.pyramid ];
698 ignoreCollisions = true;
699}
700```
701
702and running `nix-build` will create
703```
704/nix/store/cf1xhjwzmdki7fasgr4kz6di72ykicl5-python-2.7.8-env
705```
706
707with wrapped binaries in `bin/`.
708
709You can also use the `env` attribute to create local environments with needed
710packages installed. This is somewhat comparable to `virtualenv`. For example,
711running `nix-shell` with the following `shell.nix`
712```nix
713with import <nixpkgs> {};
714
715(python3.buildEnv.override {
716 extraLibs = with python3Packages; [ numpy requests ];
717}).env
718```
719
720will drop you into a shell where Python will have the
721specified packages in its path.
722
723
724##### `python.buildEnv` arguments
725
726* `extraLibs`: List of packages installed inside the environment.
727* `postBuild`: Shell command executed after the build of environment.
728* `ignoreCollisions`: Ignore file collisions inside the environment (default is `false`).
729
730#### `python.withPackages` function
731
732The `python.withPackages` function provides a simpler interface to the `python.buildEnv` functionality.
733It takes a function as an argument that is passed the set of python packages and returns the list
734of the packages to be included in the environment. Using the `withPackages` function, the previous
735example for the Pyramid Web Framework environment can be written like this:
736```nix
737with import <nixpkgs> {};
738
739python.withPackages (ps: [ps.pyramid])
740```
741
742`withPackages` passes the correct package set for the specific interpreter version as an
743argument to the function. In the above example, `ps` equals `pythonPackages`.
744But you can also easily switch to using python3:
745```nix
746with import <nixpkgs> {};
747
748python3.withPackages (ps: [ps.pyramid])
749```
750
751Now, `ps` is set to `python3Packages`, matching the version of the interpreter.
752
753As `python.withPackages` simply uses `python.buildEnv` under the hood, it also supports the `env`
754attribute. The `shell.nix` file from the previous section can thus be also written like this:
755```nix
756with import <nixpkgs> {};
757
758(python36.withPackages (ps: [ps.numpy ps.requests])).env
759```
760
761In contrast to `python.buildEnv`, `python.withPackages` does not support the more advanced options
762such as `ignoreCollisions = true` or `postBuild`. If you need them, you have to use `python.buildEnv`.
763
764Python 2 namespace packages may provide `__init__.py` that collide. In that case `python.buildEnv`
765should be used with `ignoreCollisions = true`.
766
767### Development mode
768
769Development or editable mode is supported. To develop Python packages
770`buildPythonPackage` has additional logic inside `shellPhase` to run `pip
771install -e . --prefix $TMPDIR/`for the package.
772
773Warning: `shellPhase` is executed only if `setup.py` exists.
774
775Given a `default.nix`:
776```nix
777with import <nixpkgs> {};
778
779buildPythonPackage { name = "myproject";
780
781buildInputs = with pkgs.pythonPackages; [ pyramid ];
782
783src = ./.; }
784```
785
786Running `nix-shell` with no arguments should give you
787the environment in which the package would be built with
788`nix-build`.
789
790Shortcut to setup environments with C headers/libraries and python packages:
791```shell
792nix-shell -p pythonPackages.pyramid zlib libjpeg git
793```
794
795Note: There is a boolean value `lib.inNixShell` set to `true` if nix-shell is invoked.
796
797### Tools
798
799Packages inside nixpkgs are written by hand. However many tools exist in
800community to help save time. No tool is preferred at the moment.
801
802- [python2nix](https://github.com/proger/python2nix) by Vladimir Kirillov
803- [pypi2nix](https://github.com/garbas/pypi2nix) by Rok Garbas
804- [pypi2nix](https://github.com/offlinehacker/pypi2nix) by Jaka Hudoklin
805
806### Deterministic builds
807
808Python 2.7, 3.5 and 3.6 are now built deterministically and 3.4 mostly.
809Minor modifications had to be made to the interpreters in order to generate
810deterministic bytecode. This has security implications and is relevant for
811those using Python in a `nix-shell`.
812
813When the environment variable `DETERMINISTIC_BUILD` is set, all bytecode will have timestamp 1.
814The `buildPythonPackage` function sets `DETERMINISTIC_BUILD=1` and
815[PYTHONHASHSEED=0](https://docs.python.org/3.5/using/cmdline.html#envvar-PYTHONHASHSEED).
816Both are also exported in `nix-shell`.
817
818
819### Automatic tests
820
821It is recommended to test packages as part of the build process.
822Source distributions (`sdist`) often include test files, but not always.
823
824By default the command `python setup.py test` is run as part of the
825`checkPhase`, but often it is necessary to pass a custom `checkPhase`. An
826example of such a situation is when `py.test` is used.
827
828#### Common issues
829
830- Non-working tests can often be deselected. By default `buildPythonPackage` runs `python setup.py test`.
831 Most python modules follows the standard test protocol where the pytest runner can be used instead.
832 `py.test` supports a `-k` parameter to ignore test methods or classes:
833
834 ```nix
835 buildPythonPackage {
836 # ...
837 # assumes the tests are located in tests
838 checkInputs = [ pytest ];
839 checkPhase = ''
840 py.test -k 'not function_name and not other_function' tests
841 '';
842 }
843 ```
844- Unicode issues can typically be fixed by including `glibcLocales` in `buildInputs` and exporting `LC_ALL=en_US.utf-8`.
845- Tests that attempt to access `$HOME` can be fixed by using the following work-around before running tests (e.g. `preCheck`): `export HOME=$(mktemp -d)`
846
847## FAQ
848
849### How to solve circular dependencies?
850
851Consider the packages `A` and `B` that depend on each other. When packaging `B`,
852a solution is to override package `A` not to depend on `B` as an input. The same
853should also be done when packaging `A`.
854
855### How to override a Python package?
856
857We can override the interpreter and pass `packageOverrides`.
858In the following example we rename the `pandas` package and build it.
859```nix
860with import <nixpkgs> {};
861
862(let
863 python = let
864 packageOverrides = self: super: {
865 pandas = super.pandas.overridePythonAttrs(old: {name="foo";});
866 };
867 in pkgs.python35.override {inherit packageOverrides;};
868
869in python.withPackages(ps: [ps.pandas])).env
870```
871Using `nix-build` on this expression will build an environment that contains the
872package `pandas` but with the new name `foo`.
873
874All packages in the package set will use the renamed package.
875A typical use case is to switch to another version of a certain package.
876For example, in the Nixpkgs repository we have multiple versions of `django` and `scipy`.
877In the following example we use a different version of `scipy` and create an environment that uses it.
878All packages in the Python package set will now use the updated `scipy` version.
879
880```nix
881with import <nixpkgs> {};
882
883( let
884 packageOverrides = self: super: {
885 scipy = super.scipy_0_17;
886 };
887 in (pkgs.python35.override {inherit packageOverrides;}).withPackages (ps: [ps.blaze])
888).env
889```
890The requested package `blaze` depends on `pandas` which itself depends on `scipy`.
891
892If you want the whole of Nixpkgs to use your modifications, then you can use `overlays`
893as explained in this manual. In the following example we build a `inkscape` using a different version of `numpy`.
894```nix
895let
896 pkgs = import <nixpkgs> {};
897 newpkgs = import pkgs.path { overlays = [ (pkgsself: pkgssuper: {
898 python27 = let
899 packageOverrides = self: super: {
900 numpy = super.numpy_1_10;
901 };
902 in pkgssuper.python27.override {inherit packageOverrides;};
903 } ) ]; };
904in newpkgs.inkscape
905```
906
907### `python setup.py bdist_wheel` cannot create .whl
908
909Executing `python setup.py bdist_wheel` in a `nix-shell `fails with
910```
911ValueError: ZIP does not support timestamps before 1980
912```
913
914This is because files from the Nix store (which have a timestamp of the UNIX epoch of January 1, 1970) are included in the .ZIP, but .ZIP archives follow the DOS convention of counting timestamps from 1980.
915
916The command `bdist_wheel` reads the `SOURCE_DATE_EPOCH` environment variable, which `nix-shell` sets to 1. Unsetting this variable or giving it a value corresponding to 1980 or later enables building wheels.
917
918Use 1980 as timestamp:
919```shell
920nix-shell --run "SOURCE_DATE_EPOCH=315532800 python3 setup.py bdist_wheel"
921```
922or the current time:
923```shell
924nix-shell --run "SOURCE_DATE_EPOCH=$(date +%s) python3 setup.py bdist_wheel"
925```
926or unset `SOURCE_DATE_EPOCH`:
927```shell
928nix-shell --run "unset SOURCE_DATE_EPOCH; python3 setup.py bdist_wheel"
929```
930
931### `install_data` / `data_files` problems
932
933If you get the following error:
934```
935could not create '/nix/store/6l1bvljpy8gazlsw2aw9skwwp4pmvyxw-python-2.7.8/etc':
936Permission denied
937```
938This is a [known bug](https://github.com/pypa/setuptools/issues/130) in `setuptools`.
939Setuptools `install_data` does not respect `--prefix`. An example of such package using the feature is `pkgs/tools/X11/xpra/default.nix`.
940As workaround install it as an extra `preInstall` step:
941```shell
942${python.interpreter} setup.py install_data --install-dir=$out --root=$out
943sed -i '/ = data\_files/d' setup.py
944```
945
946### Rationale of non-existent global site-packages
947
948On most operating systems a global `site-packages` is maintained. This however
949becomes problematic if you want to run multiple Python versions or have multiple
950versions of certain libraries for your projects. Generally, you would solve such
951issues by creating virtual environments using `virtualenv`.
952
953On Nix each package has an isolated dependency tree which, in the case of
954Python, guarantees the right versions of the interpreter and libraries or
955packages are available. There is therefore no need to maintain a global `site-packages`.
956
957If you want to create a Python environment for development, then the recommended
958method is to use `nix-shell`, either with or without the `python.buildEnv`
959function.
960
961### How to consume python modules using pip in a virtualenv like I am used to on other Operating Systems ?
962
963This is an example of a `default.nix` for a `nix-shell`, which allows to consume a `virtualenv` environment,
964and install python modules through `pip` the traditional way.
965
966Create this `default.nix` file, together with a `requirements.txt` and simply execute `nix-shell`.
967
968```nix
969with import <nixpkgs> {};
970with pkgs.python27Packages;
971
972stdenv.mkDerivation {
973 name = "impurePythonEnv";
974 buildInputs = [
975 # these packages are required for virtualenv and pip to work:
976 #
977 python27Full
978 python27Packages.virtualenv
979 python27Packages.pip
980 # the following packages are related to the dependencies of your python
981 # project.
982 # In this particular example the python modules listed in the
983 # requirements.txt require the following packages to be installed locally
984 # in order to compile any binary extensions they may require.
985 #
986 taglib
987 openssl
988 git
989 libxml2
990 libxslt
991 libzip
992 stdenv
993 zlib ];
994 src = null;
995 shellHook = ''
996 # set SOURCE_DATE_EPOCH so that we can use python wheels
997 SOURCE_DATE_EPOCH=$(date +%s)
998 virtualenv --no-setuptools venv
999 export PATH=$PWD/venv/bin:$PATH
1000 pip install -r requirements.txt
1001 '';
1002}
1003```
1004
1005Note that the `pip install` is an imperative action. So every time `nix-shell`
1006is executed it will attempt to download the python modules listed in
1007requirements.txt. However these will be cached locally within the `virtualenv`
1008folder and not downloaded again.
1009
1010### How to override a Python package from `configuration.nix`?
1011
1012If you need to change a package's attribute(s) from `configuration.nix` you could do:
1013
1014```nix
1015 nixpkgs.config.packageOverrides = super: {
1016 python = super.python.override {
1017 packageOverrides = python-self: python-super: {
1018 zerobin = python-super.zerobin.overrideAttrs (oldAttrs: {
1019 src = super.fetchgit {
1020 url = "https://github.com/sametmax/0bin";
1021 rev = "a344dbb18fe7a855d0742b9a1cede7ce423b34ec";
1022 sha256 = "16d769kmnrpbdr0ph0whyf4yff5df6zi4kmwx7sz1d3r6c8p6xji";
1023 };
1024 });
1025 };
1026 };
1027 };
1028```
1029
1030`pythonPackages.zerobin` is now globally overridden. All packages and also the
1031`zerobin` NixOS service use the new definition.
1032Note that `python-super` refers to the old package set and `python-self`
1033to the new, overridden version.
1034
1035To modify only a Python package set instead of a whole Python derivation, use this snippet:
1036
1037```nix
1038 myPythonPackages = pythonPackages.override {
1039 overrides = self: super: {
1040 zerobin = ...;
1041 };
1042 }
1043```
1044
1045### How to override a Python package using overlays?
1046
1047Use the following overlay template:
1048
1049```nix
1050self: super:
1051{
1052 python = super.python.override {
1053 packageOverrides = python-self: python-super: {
1054 zerobin = python-super.zerobin.overrideAttrs (oldAttrs: {
1055 src = super.fetchgit {
1056 url = "https://github.com/sametmax/0bin";
1057 rev = "a344dbb18fe7a855d0742b9a1cede7ce423b34ec";
1058 sha256 = "16d769kmnrpbdr0ph0whyf4yff5df6zi4kmwx7sz1d3r6c8p6xji";
1059 };
1060 });
1061 };
1062 };
1063}
1064```
1065
1066## Contributing
1067
1068### Contributing guidelines
1069
1070Following rules are desired to be respected:
1071
1072* Python libraries are called from `python-packages.nix` and packaged with `buildPythonPackage`. The expression of a library should be in `pkgs/development/python-modules/<name>/default.nix`. Libraries in `pkgs/top-level/python-packages.nix` are sorted quasi-alphabetically to avoid merge conflicts.
1073* Python applications live outside of `python-packages.nix` and are packaged with `buildPythonApplication`.
1074* Make sure libraries build for all Python interpreters.
1075* By default we enable tests. Make sure the tests are found and, in the case of libraries, are passing for all interpreters. If certain tests fail they can be disabled individually. Try to avoid disabling the tests altogether. In any case, when you disable tests, leave a comment explaining why.
1076* Commit names of Python libraries should reflect that they are Python libraries, so write for example `pythonPackages.numpy: 1.11 -> 1.12`.
1077* Attribute names in `python-packages.nix` should be normalized according to [PEP 0503](https://www.python.org/dev/peps/pep-0503/#normalized-names).
1078 This means that characters should be converted to lowercase and `.` and `_` should be replaced by a single `-` (foo-bar-baz instead of Foo__Bar.baz )