pyrox.dev
1# Python
2
3## User Guide
4
5### Using Python
6
7#### Overview
8
9Several versions of the Python interpreter are available on Nix, as well as a
10high amount of packages. The attribute `python` refers to the default
11interpreter, which is currently CPython 2.7. It is also possible to refer to
12specific versions, e.g. `python35` refers to CPython 3.5, and `pypy` refers to
13the default PyPy interpreter.
14
15Python is used a lot, and in different ways. This affects also how it is
16packaged. In the case of Python on Nix, an important distinction is made between
17whether the package is considered primarily an application, or whether it should
18be used as a library, i.e., of primary interest are the modules in
19`site-packages` that should be importable.
20
21In the Nixpkgs tree Python applications can be found throughout, depending on
22what they do, and are called from the main package set. Python libraries,
23however, are in separate sets, with one set per interpreter version.
24
25The interpreters have several common attributes. One of these attributes is
26`pkgs`, which is a package set of Python libraries for this specific
27interpreter. E.g., the `toolz` package corresponding to the default interpreter
28is `python.pkgs.toolz`, and the CPython 3.5 version is `python35.pkgs.toolz`.
29The main package set contains aliases to these package sets, e.g.
30`pythonPackages` refers to `python.pkgs` and `python35Packages` to
31`python35.pkgs`.
32
33#### Installing Python and packages
34
35The Nix and NixOS manuals explain how packages are generally installed. In the
36case of Python and Nix, it is important to make a distinction between whether the
37package is considered an application or a library.
38
39Applications on Nix are typically installed into your user
40profile imperatively using `nix-env -i`, and on NixOS declaratively by adding the
41package name to `environment.systemPackages` in `/etc/nixos/configuration.nix`.
42Dependencies such as libraries are automatically installed and should not be
43installed explicitly.
44
45The same goes for Python applications and libraries. Python applications can be
46installed in your profile. But Python libraries you would like to use for
47development cannot be installed, at least not individually, because they won't
48be able to find each other resulting in import errors. Instead, it is possible
49to create an environment with `python.buildEnv` or `python.withPackages` where
50the interpreter and other executables are able to find each other and all of the
51modules.
52
53In the following examples we create an environment with Python 3.5, `numpy` and
54`toolz`. As you may imagine, there is one limitation here, and that's that
55you can install only one environment at a time. You will notice the complaints
56about collisions when you try to install a second environment.
57
58##### Environment defined in separate `.nix` file
59
60Create a file, e.g. `build.nix`, with the following expression
61```nix
62with import <nixpkgs> {};
63
64python35.withPackages (ps: with ps; [ numpy toolz ])
65```
66and install it in your profile with
67```shell
68nix-env -if build.nix
69```
70Now you can use the Python interpreter, as well as the extra packages (`numpy`,
71`toolz`) that you added to the environment.
72
73##### Environment defined in `~/.config/nixpkgs/config.nix`
74
75If you prefer to, you could also add the environment as a package override to the Nixpkgs set, e.g.
76using `config.nix`,
77```nix
78{ # ...
79
80 packageOverrides = pkgs: with pkgs; {
81 myEnv = python35.withPackages (ps: with ps; [ numpy toolz ]);
82 };
83}
84```
85and install it in your profile with
86```shell
87nix-env -iA nixpkgs.myEnv
88```
89The environment is is installed by referring to the attribute, and considering
90the `nixpkgs` channel was used.
91
92##### Environment defined in `/etc/nixos/configuration.nix`
93
94For the sake of completeness, here's another example how to install the environment system-wide.
95
96```nix
97{ # ...
98
99 environment.systemPackages = with pkgs; [
100 (python35.withPackages(ps: with ps; [ numpy toolz ]))
101 ];
102}
103```
104
105#### Temporary Python environment with `nix-shell`
106
107The examples in the previous section showed how to install a Python environment
108into a profile. For development you may need to use multiple environments.
109`nix-shell` gives the possibility to temporarily load another environment, akin
110to `virtualenv`.
111
112There are two methods for loading a shell with Python packages. The first and recommended method
113is to create an environment with `python.buildEnv` or `python.withPackages` and load that. E.g.
114```sh
115$ nix-shell -p 'python35.withPackages(ps: with ps; [ numpy toolz ])'
116```
117opens a shell from which you can launch the interpreter
118```sh
119[nix-shell:~] python3
120```
121The other method, which is not recommended, does not create an environment and requires you to list the packages directly,
122
123```sh
124$ nix-shell -p python35.pkgs.numpy python35.pkgs.toolz
125```
126Again, it is possible to launch the interpreter from the shell.
127The Python interpreter has the attribute `pkgs` which contains all Python libraries for that specific interpreter.
128
129##### Load environment from `.nix` expression
130As explained in the Nix manual, `nix-shell` can also load an
131expression from a `.nix` file. Say we want to have Python 3.5, `numpy`
132and `toolz`, like before, in an environment. Consider a `shell.nix` file
133with
134```nix
135with import <nixpkgs> {};
136
137(python35.withPackages (ps: [ps.numpy ps.toolz])).env
138```
139Executing `nix-shell` gives you again a Nix shell from which you can run Python.
140
141What's happening here?
142
1431. We begin with importing the Nix Packages collections. `import <nixpkgs>` imports the `<nixpkgs>` function, `{}` calls it and the `with` statement brings all attributes of `nixpkgs` in the local scope. These attributes form the main package set.
1442. Then we create a Python 3.5 environment with the `withPackages` function.
1453. The `withPackages` function expects us to provide a function as an argument that takes the set of all python packages and returns a list of packages to include in the environment. Here, we select the packages `numpy` and `toolz` from the package set.
146
147##### Execute command with `--run`
148A convenient option with `nix-shell` is the `--run`
149option, with which you can execute a command in the `nix-shell`. We can
150e.g. directly open a Python shell
151```sh
152$ nix-shell -p python35Packages.numpy python35Packages.toolz --run "python3"
153```
154or run a script
155```sh
156$ nix-shell -p python35Packages.numpy python35Packages.toolz --run "python3 myscript.py"
157```
158
159##### `nix-shell` as shebang
160In fact, for the second use case, there is a more convenient method. You can
161add a [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) to your script
162specifying which dependencies `nix-shell` needs. With the following shebang, you
163can just execute `./myscript.py`, and it will make available all dependencies and
164run the script in the `python3` shell.
165
166```py
167#! /usr/bin/env nix-shell
168#! nix-shell -i python3 -p "python3.withPackages(ps: [ps.numpy])"
169
170import numpy
171
172print(numpy.__version__)
173```
174
175### Developing with Python
176
177Now that you know how to get a working Python environment with Nix, it is time
178to go forward and start actually developing with Python. We will first have a
179look at how Python packages are packaged on Nix. Then, we will look at how you
180can use development mode with your code.
181
182#### Packaging a library
183
184With Nix all packages are built by functions. The main function in Nix for
185building Python libraries is `buildPythonPackage`. Let's see how we can build the
186`toolz` package.
187
188```nix
189{ # ...
190
191 toolz = buildPythonPackage rec {
192 pname = "toolz";
193 version = "0.7.4";
194
195 src = fetchPypi {
196 inherit pname version;
197 sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed05afd";
198 };
199
200 doCheck = false;
201
202 meta = {
203 homepage = "http://github.com/pytoolz/toolz/";
204 description = "List processing tools and functional utilities";
205 license = licenses.bsd3;
206 maintainers = with maintainers; [ fridh ];
207 };
208 };
209}
210```
211
212What happens here? The function `buildPythonPackage` is called and as argument
213it accepts a set. In this case the set is a recursive set, `rec`. One of the
214arguments is the name of the package, which consists of a basename (generally
215following the name on PyPi) and a version. Another argument, `src` specifies the
216source, which in this case is fetched from PyPI using the helper function
217`fetchPypi`. The argument `doCheck` is used to set whether tests should be run
218when building the package. Furthermore, we specify some (optional) meta
219information. The output of the function is a derivation.
220
221An expression for `toolz` can be found in the Nixpkgs repository. As explained
222in the introduction of this Python section, a derivation of `toolz` is available
223for each interpreter version, e.g. `python35.pkgs.toolz` refers to the `toolz`
224derivation corresponding to the CPython 3.5 interpreter.
225The above example works when you're directly working on
226`pkgs/top-level/python-packages.nix` in the Nixpkgs repository. Often though,
227you will want to test a Nix expression outside of the Nixpkgs tree.
228
229The following expression creates a derivation for the `toolz` package,
230and adds it along with a `numpy` package to a Python environment.
231
232```nix
233with import <nixpkgs> {};
234
235( let
236 my_toolz = python35.pkgs.buildPythonPackage rec {
237 pname = "toolz";
238 version = "0.7.4";
239
240 src = python35.pkgs.fetchPypi {
241 inherit pname version;
242 sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed05afd";
243 };
244
245 doCheck = false;
246
247 meta = {
248 homepage = "http://github.com/pytoolz/toolz/";
249 description = "List processing tools and functional utilities";
250 };
251 };
252
253 in python35.withPackages (ps: [ps.numpy my_toolz])
254).env
255```
256Executing `nix-shell` will result in an environment in which you can use
257Python 3.5 and the `toolz` package. As you can see we had to explicitly mention
258for which Python version we want to build a package.
259
260So, what did we do here? Well, we took the Nix expression that we used earlier
261to build a Python environment, and said that we wanted to include our own
262version of `toolz`, named `my_toolz`. To introduce our own package in the scope
263of `withPackages` we used a `let` expression. You can see that we used
264`ps.numpy` to select numpy from the nixpkgs package set (`ps`). We did not take
265`toolz` from the Nixpkgs package set this time, but instead took our own version
266that we introduced with the `let` expression.
267
268#### Handling dependencies
269
270Our example, `toolz`, does not have any dependencies on other Python
271packages or system libraries. According to the manual, `buildPythonPackage`
272uses the arguments `buildInputs` and `propagatedBuildInputs` to specify dependencies. If something is
273exclusively a build-time dependency, then the dependency should be included as a
274`buildInput`, but if it is (also) a runtime dependency, then it should be added
275to `propagatedBuildInputs`. Test dependencies are considered build-time dependencies.
276
277The following example shows which arguments are given to `buildPythonPackage` in
278order to build [`datashape`](https://github.com/blaze/datashape).
279
280```nix
281{ # ...
282
283 datashape = buildPythonPackage rec {
284 pname = "datashape";
285 version = "0.4.7";
286
287 src = fetchPypi {
288 inherit pname version;
289 sha256 = "14b2ef766d4c9652ab813182e866f493475e65e558bed0822e38bf07bba1a278";
290 };
291
292 checkInputs = with self; [ pytest ];
293 propagatedBuildInputs = with self; [ numpy multipledispatch dateutil ];
294
295 meta = {
296 homepage = https://github.com/ContinuumIO/datashape;
297 description = "A data description language";
298 license = licenses.bsd2;
299 maintainers = with maintainers; [ fridh ];
300 };
301 };
302}
303```
304
305We can see several runtime dependencies, `numpy`, `multipledispatch`, and
306`dateutil`. Furthermore, we have one `buildInput`, i.e. `pytest`. `pytest` is a
307test runner and is only used during the `checkPhase` and is therefore not added
308to `propagatedBuildInputs`.
309
310In the previous case we had only dependencies on other Python packages to consider.
311Occasionally you have also system libraries to consider. E.g., `lxml` provides
312Python bindings to `libxml2` and `libxslt`. These libraries are only required
313when building the bindings and are therefore added as `buildInputs`.
314
315```nix
316{ # ...
317
318 lxml = buildPythonPackage rec {
319 pname = "lxml";
320 version = "3.4.4";
321
322 src = fetchPypi {
323 inherit pname version;
324 sha256 = "16a0fa97hym9ysdk3rmqz32xdjqmy4w34ld3rm3jf5viqjx65lxk";
325 };
326
327 buildInputs = with self; [ pkgs.libxml2 pkgs.libxslt ];
328
329 meta = {
330 description = "Pythonic binding for the libxml2 and libxslt libraries";
331 homepage = http://lxml.de;
332 license = licenses.bsd3;
333 maintainers = with maintainers; [ sjourdois ];
334 };
335 };
336}
337```
338
339In this example `lxml` and Nix are able to work out exactly where the relevant
340files of the dependencies are. This is not always the case.
341
342The example below shows bindings to The Fastest Fourier Transform in the West, commonly known as
343FFTW. On Nix we have separate packages of FFTW for the different types of floats
344(`"single"`, `"double"`, `"long-double"`). The bindings need all three types,
345and therefore we add all three as `buildInputs`. The bindings don't expect to
346find each of them in a different folder, and therefore we have to set `LDFLAGS`
347and `CFLAGS`.
348
349```nix
350{ # ...
351
352 pyfftw = buildPythonPackage rec {
353 pname = "pyFFTW";
354 version = "0.9.2";
355
356 src = fetchPypi {
357 inherit pname version;
358 sha256 = "f6bbb6afa93085409ab24885a1a3cdb8909f095a142f4d49e346f2bd1b789074";
359 };
360
361 buildInputs = [ pkgs.fftw pkgs.fftwFloat pkgs.fftwLongDouble];
362
363 propagatedBuildInputs = with self; [ numpy scipy ];
364
365 # Tests cannot import pyfftw. pyfftw works fine though.
366 doCheck = false;
367
368 preConfigure = ''
369 export LDFLAGS="-L${pkgs.fftw.dev}/lib -L${pkgs.fftwFloat.out}/lib -L${pkgs.fftwLongDouble.out}/lib"
370 export CFLAGS="-I${pkgs.fftw.dev}/include -I${pkgs.fftwFloat.dev}/include -I${pkgs.fftwLongDouble.dev}/include"
371 '';
372
373 meta = {
374 description = "A pythonic wrapper around FFTW, the FFT library, presenting a unified interface for all the supported transforms";
375 homepage = http://hgomersall.github.com/pyFFTW/;
376 license = with licenses; [ bsd2 bsd3 ];
377 maintainer = with maintainers; [ fridh ];
378 };
379 };
380}
381```
382Note also the line `doCheck = false;`, we explicitly disabled running the test-suite.
383
384
385#### Develop local package
386
387As a Python developer you're likely aware of [development mode](http://setuptools.readthedocs.io/en/latest/setuptools.html#development-mode) (`python setup.py develop`);
388instead of installing the package this command creates a special link to the project code.
389That way, you can run updated code without having to reinstall after each and every change you make.
390Development mode is also available. Let's see how you can use it.
391
392In the previous Nix expression the source was fetched from an url. We can also refer to a local source instead using
393`src = ./path/to/source/tree;`
394
395If we create a `shell.nix` file which calls `buildPythonPackage`, and if `src`
396is a local source, and if the local source has a `setup.py`, then development
397mode is activated.
398
399In the following example we create a simple environment that
400has a Python 3.5 version of our package in it, as well as its dependencies and
401other packages we like to have in the environment, all specified with `propagatedBuildInputs`.
402Indeed, we can just add any package we like to have in our environment to `propagatedBuildInputs`.
403
404```nix
405with import <nixpkgs> {};
406with pkgs.python35Packages;
407
408buildPythonPackage rec {
409 name = "mypackage";
410 src = ./path/to/package/source;
411 propagatedBuildInputs = [ pytest numpy pkgs.libsndfile ];
412}
413```
414
415It is important to note that due to how development mode is implemented on Nix it is not possible to have multiple packages simultaneously in development mode.
416
417
418### Organising your packages
419
420So far we discussed how you can use Python on Nix, and how you can develop with
421it. We've looked at how you write expressions to package Python packages, and we
422looked at how you can create environments in which specified packages are
423available.
424
425At some point you'll likely have multiple packages which you would
426like to be able to use in different projects. In order to minimise unnecessary
427duplication we now look at how you can maintain yourself a repository with your
428own packages. The important functions here are `import` and `callPackage`.
429
430### Including a derivation using `callPackage`
431
432Earlier we created a Python environment using `withPackages`, and included the
433`toolz` package via a `let` expression.
434Let's split the package definition from the environment definition.
435
436We first create a function that builds `toolz` in `~/path/to/toolz/release.nix`
437
438```nix
439{ pkgs, buildPythonPackage }:
440
441buildPythonPackage rec {
442 pname = "toolz";
443 version = "0.7.4";
444
445 src = fetchPypi {
446 inherit pname version;
447 sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed05afd";
448 };
449
450 meta = {
451 homepage = "http://github.com/pytoolz/toolz/";
452 description = "List processing tools and functional utilities";
453 license = licenses.bsd3;
454 maintainers = with maintainers; [ fridh ];
455 };
456}
457```
458
459It takes two arguments, `pkgs` and `buildPythonPackage`.
460We now call this function using `callPackage` in the definition of our environment
461
462```nix
463with import <nixpkgs> {};
464
465( let
466 toolz = pkgs.callPackage /path/to/toolz/release.nix {
467 pkgs = pkgs;
468 buildPythonPackage = pkgs.python35Packages.buildPythonPackage;
469 };
470 in pkgs.python35.withPackages (ps: [ ps.numpy toolz ])
471).env
472```
473
474Important to remember is that the Python version for which the package is made
475depends on the `python` derivation that is passed to `buildPythonPackage`. Nix
476tries to automatically pass arguments when possible, which is why generally you
477don't explicitly define which `python` derivation should be used. In the above
478example we use `buildPythonPackage` that is part of the set `python35Packages`,
479and in this case the `python35` interpreter is automatically used.
480
481
482
483## Reference
484
485### Interpreters
486
487Versions 2.7, 3.3, 3.4, 3.5 and 3.6 of the CPython interpreter are available as
488respectively `python27`, `python34`, `python35` and `python36`. The PyPy interpreter
489is available as `pypy`. The aliases `python2` and `python3` correspond to respectively `python27` and
490`python35`. The default interpreter, `python`, maps to `python2`.
491The Nix expressions for the interpreters can be found in
492`pkgs/development/interpreters/python`.
493
494All packages depending on any Python interpreter get appended
495`out/{python.sitePackages}` to `$PYTHONPATH` if such directory
496exists.
497
498#### Missing `tkinter` module standard library
499
500To reduce closure size the `Tkinter`/`tkinter` is available as a separate package, `pythonPackages.tkinter`.
501
502#### Attributes on interpreters packages
503
504Each interpreter has the following attributes:
505
506- `libPrefix`. Name of the folder in `${python}/lib/` for corresponding interpreter.
507- `interpreter`. Alias for `${python}/bin/${executable}`.
508- `buildEnv`. Function to build python interpreter environments with extra packages bundled together. See section *python.buildEnv function* for usage and documentation.
509- `withPackages`. Simpler interface to `buildEnv`. See section *python.withPackages function* for usage and documentation.
510- `sitePackages`. Alias for `lib/${libPrefix}/site-packages`.
511- `executable`. Name of the interpreter executable, e.g. `python3.4`.
512- `pkgs`. Set of Python packages for that specific interpreter. The package set can be modified by overriding the interpreter and passing `packageOverrides`.
513
514### Building packages and applications
515
516Python libraries and applications that use `setuptools` or
517`distutils` are typically build with respectively the `buildPythonPackage` and
518`buildPythonApplication` functions. These two functions also support installing a `wheel`.
519
520All Python packages reside in `pkgs/top-level/python-packages.nix` and all
521applications elsewhere. In case a package is used as both a library and an application,
522then the package should be in `pkgs/top-level/python-packages.nix` since only those packages are made
523available for all interpreter versions. The preferred location for library expressions is in
524`pkgs/development/python-modules`. It is important that these packages are
525called from `pkgs/top-level/python-packages.nix` and not elsewhere, to guarantee
526the right version of the package is built.
527
528Based on the packages defined in `pkgs/top-level/python-packages.nix` an
529attribute set is created for each available Python interpreter. The available
530sets are
531
532* `pkgs.python27Packages`
533* `pkgs.python34Packages`
534* `pkgs.python35Packages`
535* `pkgs.python36Packages`
536* `pkgs.pypyPackages`
537
538and the aliases
539
540* `pkgs.python2Packages` pointing to `pkgs.python27Packages`
541* `pkgs.python3Packages` pointing to `pkgs.python36Packages`
542* `pkgs.pythonPackages` pointing to `pkgs.python2Packages`
543
544#### `buildPythonPackage` function
545
546The `buildPythonPackage` function is implemented in
547`pkgs/development/interpreters/python/build-python-package.nix`
548
549The following is an example:
550```nix
551
552buildPythonPackage rec {
553 version = "3.3.1";
554 pname = "pytest";
555
556 preCheck = ''
557 # don't test bash builtins
558 rm testing/test_argcomplete.py
559 '';
560
561 src = fetchPypi {
562 inherit pname version;
563 sha256 = "cf8436dc59d8695346fcd3ab296de46425ecab00d64096cebe79fb51ecb2eb93";
564 };
565
566 checkInputs = [ hypothesis ];
567 buildInputs = [ setuptools_scm ];
568 propagatedBuildInputs = [ attrs py setuptools six pluggy ];
569
570 meta = with stdenv.lib; {
571 maintainers = with maintainers; [ domenkozar lovek323 madjar lsix ];
572 description = "Framework for writing tests";
573 };
574}
575
576```
577
578The `buildPythonPackage` mainly does four things:
579
580* In the `buildPhase`, it calls `${python.interpreter} setup.py bdist_wheel` to
581 build a wheel binary zipfile.
582* In the `installPhase`, it installs the wheel file using `pip install *.whl`.
583* In the `postFixup` phase, the `wrapPythonPrograms` bash function is called to
584 wrap all programs in the `$out/bin/*` directory to include `$PATH`
585 environment variable and add dependent libraries to script's `sys.path`.
586* In the `installCheck` phase, `${python.interpreter} setup.py test` is ran.
587
588As in Perl, dependencies on other Python packages can be specified in the
589`buildInputs` and `propagatedBuildInputs` attributes. If something is
590exclusively a build-time dependency, use `buildInputs`; if it’s (also) a runtime
591dependency, use `propagatedBuildInputs`.
592
593By default tests are run because `doCheck = true`. Test dependencies, like
594e.g. the test runner, should be added to `buildInputs`.
595
596By default `meta.platforms` is set to the same value
597as the interpreter unless overridden otherwise.
598
599##### `buildPythonPackage` parameters
600
601All parameters from `mkDerivation` function are still supported.
602
603* `namePrefix`: Prepended text to `${name}` parameter. Defaults to `"python3.3-"` for Python 3.3, etc. Set it to `""` if you're packaging an application or a command line tool.
604* `disabled`: If `true`, package is not build for particular python interpreter version. Grep around `pkgs/top-level/python-packages.nix` for examples.
605* `setupPyBuildFlags`: List of flags passed to `setup.py build_ext` command.
606* `pythonPath`: List of packages to be added into `$PYTHONPATH`. Packages in `pythonPath` are not propagated (contrary to `propagatedBuildInputs`).
607* `preShellHook`: Hook to execute commands before `shellHook`.
608* `postShellHook`: Hook to execute commands after `shellHook`.
609* `makeWrapperArgs`: A list of strings. Arguments to be passed to `makeWrapper`, which wraps generated binaries. By default, the arguments to `makeWrapper` set `PATH` and `PYTHONPATH` environment variables before calling the binary. Additional arguments here can allow a developer to set environment variables which will be available when the binary is run. For example, `makeWrapperArgs = ["--set FOO BAR" "--set BAZ QUX"]`.
610* `installFlags`: A list of strings. Arguments to be passed to `pip install`. To pass options to `python setup.py install`, use `--install-option`. E.g., `installFlags=["--install-option='--cpp_implementation'"].
611* `format`: Format of the source. Valid options are `setuptools` (default), `flit`, `wheel`, and `other`. `setuptools` is for when the source has a `setup.py` and `setuptools` is used to build a wheel, `flit`, in case `flit` should be used to build a wheel, and `wheel` in case a wheel is provided. In case you need to provide your own `buildPhase` and `installPhase` you can use `other`.
612* `catchConflicts` If `true`, abort package build if a package name appears more than once in dependency tree. Default is `true`.
613* `checkInputs` Dependencies needed for running the `checkPhase`. These are added to `buildInputs` when `doCheck = true`.
614
615##### Overriding Python packages
616
617The `buildPythonPackage` function has a `overridePythonAttrs` method that
618can be used to override the package. In the following example we create an
619environment where we have the `blaze` package using an older version of `pandas`.
620We override first the Python interpreter and pass
621`packageOverrides` which contains the overrides for packages in the package set.
622
623```nix
624with import <nixpkgs> {};
625
626(let
627 python = let
628 packageOverrides = self: super: {
629 pandas = super.pandas.overridePythonAttrs(old: rec {
630 version = "0.19.1";
631 src = super.fetchPypi {
632 pname = "pandas";
633 inherit version;
634 sha256 = "08blshqj9zj1wyjhhw3kl2vas75vhhicvv72flvf1z3jvapgw295";
635 };
636 });
637 };
638 in pkgs.python3.override {inherit packageOverrides;};
639
640in python.withPackages(ps: [ps.blaze])).env
641```
642
643#### `buildPythonApplication` function
644
645The `buildPythonApplication` function is practically the same as `buildPythonPackage`.
646The difference is that `buildPythonPackage` by default prefixes the names of the packages with the version of the interpreter.
647Because with an application we're not interested in multiple version the prefix is dropped.
648
649#### python.buildEnv function
650
651Python environments can be created using the low-level `pkgs.buildEnv` function.
652This example shows how to create an environment that has the Pyramid Web Framework.
653Saving the following as `default.nix`
654```nix
655with import <nixpkgs> {};
656
657python.buildEnv.override {
658 extraLibs = [ pkgs.pythonPackages.pyramid ];
659 ignoreCollisions = true;
660}
661```
662
663and running `nix-build` will create
664```
665/nix/store/cf1xhjwzmdki7fasgr4kz6di72ykicl5-python-2.7.8-env
666```
667
668with wrapped binaries in `bin/`.
669
670You can also use the `env` attribute to create local environments with needed
671packages installed. This is somewhat comparable to `virtualenv`. For example,
672running `nix-shell` with the following `shell.nix`
673```nix
674with import <nixpkgs> {};
675
676(python3.buildEnv.override {
677 extraLibs = with python3Packages; [ numpy requests ];
678}).env
679```
680
681will drop you into a shell where Python will have the
682specified packages in its path.
683
684
685##### `python.buildEnv` arguments
686
687* `extraLibs`: List of packages installed inside the environment.
688* `postBuild`: Shell command executed after the build of environment.
689* `ignoreCollisions`: Ignore file collisions inside the environment (default is `false`).
690
691#### python.withPackages function
692
693The `python.withPackages` function provides a simpler interface to the `python.buildEnv` functionality.
694It takes a function as an argument that is passed the set of python packages and returns the list
695of the packages to be included in the environment. Using the `withPackages` function, the previous
696example for the Pyramid Web Framework environment can be written like this:
697```nix
698with import <nixpkgs> {};
699
700python.withPackages (ps: [ps.pyramid])
701```
702
703`withPackages` passes the correct package set for the specific interpreter version as an
704argument to the function. In the above example, `ps` equals `pythonPackages`.
705But you can also easily switch to using python3:
706```nix
707with import <nixpkgs> {};
708
709python3.withPackages (ps: [ps.pyramid])
710```
711
712Now, `ps` is set to `python3Packages`, matching the version of the interpreter.
713
714As `python.withPackages` simply uses `python.buildEnv` under the hood, it also supports the `env`
715attribute. The `shell.nix` file from the previous section can thus be also written like this:
716```nix
717with import <nixpkgs> {};
718
719(python36.withPackages (ps: [ps.numpy ps.requests])).env
720```
721
722In contrast to `python.buildEnv`, `python.withPackages` does not support the more advanced options
723such as `ignoreCollisions = true` or `postBuild`. If you need them, you have to use `python.buildEnv`.
724
725Python 2 namespace packages may provide `__init__.py` that collide. In that case `python.buildEnv`
726should be used with `ignoreCollisions = true`.
727
728### Development mode
729
730Development or editable mode is supported. To develop Python packages
731`buildPythonPackage` has additional logic inside `shellPhase` to run `pip
732install -e . --prefix $TMPDIR/`for the package.
733
734Warning: `shellPhase` is executed only if `setup.py` exists.
735
736Given a `default.nix`:
737```nix
738with import <nixpkgs> {};
739
740buildPythonPackage { name = "myproject";
741
742buildInputs = with pkgs.pythonPackages; [ pyramid ];
743
744src = ./.; }
745```
746
747Running `nix-shell` with no arguments should give you
748the environment in which the package would be built with
749`nix-build`.
750
751Shortcut to setup environments with C headers/libraries and python packages:
752```shell
753nix-shell -p pythonPackages.pyramid zlib libjpeg git
754```
755
756Note: There is a boolean value `lib.inNixShell` set to `true` if nix-shell is invoked.
757
758### Tools
759
760Packages inside nixpkgs are written by hand. However many tools exist in
761community to help save time. No tool is preferred at the moment.
762
763- [python2nix](https://github.com/proger/python2nix) by Vladimir Kirillov
764- [pypi2nix](https://github.com/garbas/pypi2nix) by Rok Garbas
765- [pypi2nix](https://github.com/offlinehacker/pypi2nix) by Jaka Hudoklin
766
767### Deterministic builds
768
769Python 2.7, 3.5 and 3.6 are now built deterministically and 3.4 mostly.
770Minor modifications had to be made to the interpreters in order to generate
771deterministic bytecode. This has security implications and is relevant for
772those using Python in a `nix-shell`.
773
774When the environment variable `DETERMINISTIC_BUILD` is set, all bytecode will have timestamp 1.
775The `buildPythonPackage` function sets `DETERMINISTIC_BUILD=1` and
776[PYTHONHASHSEED=0](https://docs.python.org/3.5/using/cmdline.html#envvar-PYTHONHASHSEED).
777Both are also exported in `nix-shell`.
778
779
780### Automatic tests
781
782It is recommended to test packages as part of the build process.
783Source distributions (`sdist`) often include test files, but not always.
784
785By default the command `python setup.py test` is run as part of the
786`checkPhase`, but often it is necessary to pass a custom `checkPhase`. An
787example of such a situation is when `py.test` is used.
788
789#### Common issues
790
791- Non-working tests can often be deselected. By default `buildPythonPackage` runs `python setup.py test`.
792 Most python modules follows the standard test protocol where the pytest runner can be used instead.
793 `py.test` supports a `-k` parameter to ignore test methods or classes:
794
795 ```nix
796 buildPythonPackage {
797 # ...
798 # assumes the tests are located in tests
799 checkInputs = [ pytest ];
800 checkPhase = ''
801 py.test -k 'not function_name and not other_function' tests
802 '';
803 }
804 ```
805- Unicode issues can typically be fixed by including `glibcLocales` in `buildInputs` and exporting `LC_ALL=en_US.utf-8`.
806- Tests that attempt to access `$HOME` can be fixed by using the following work-around before running tests (e.g. `preCheck`): `export HOME=$(mktemp -d)`
807
808## FAQ
809
810### How to solve circular dependencies?
811
812Consider the packages `A` and `B` that depend on each other. When packaging `B`,
813a solution is to override package `A` not to depend on `B` as an input. The same
814should also be done when packaging `A`.
815
816### How to override a Python package?
817
818We can override the interpreter and pass `packageOverrides`.
819In the following example we rename the `pandas` package and build it.
820```nix
821with import <nixpkgs> {};
822
823(let
824 python = let
825 packageOverrides = self: super: {
826 pandas = super.pandas.overridePythonAttrs(old: {name="foo";});
827 };
828 in pkgs.python35.override {inherit packageOverrides;};
829
830in python.withPackages(ps: [ps.pandas])).env
831```
832Using `nix-build` on this expression will build an environment that contains the
833package `pandas` but with the new name `foo`.
834
835All packages in the package set will use the renamed package.
836A typical use case is to switch to another version of a certain package.
837For example, in the Nixpkgs repository we have multiple versions of `django` and `scipy`.
838In the following example we use a different version of `scipy` and create an environment that uses it.
839All packages in the Python package set will now use the updated `scipy` version.
840
841```nix
842with import <nixpkgs> {};
843
844( let
845 packageOverrides = self: super: {
846 scipy = super.scipy_0_17;
847 };
848 in (pkgs.python35.override {inherit packageOverrides;}).withPackages (ps: [ps.blaze])
849).env
850```
851The requested package `blaze` depends on `pandas` which itself depends on `scipy`.
852
853If you want the whole of Nixpkgs to use your modifications, then you can use `overlays`
854as explained in this manual. In the following example we build a `inkscape` using a different version of `numpy`.
855```nix
856let
857 pkgs = import <nixpkgs> {};
858 newpkgs = import pkgs.path { overlays = [ (pkgsself: pkgssuper: {
859 python27 = let
860 packageOverrides = self: super: {
861 numpy = super.numpy_1_10;
862 };
863 in pkgssuper.python27.override {inherit packageOverrides;};
864 } ) ]; };
865in newpkgs.inkscape
866```
867
868### `python setup.py bdist_wheel` cannot create .whl
869
870Executing `python setup.py bdist_wheel` in a `nix-shell `fails with
871```
872ValueError: ZIP does not support timestamps before 1980
873```
874This is because files are included that depend on items in the Nix store which have a timestamp of, that is, it corresponds to January the 1st, 1970 at 00:00:00. And as the error informs you, ZIP does not support that.
875The command `bdist_wheel` takes into account `SOURCE_DATE_EPOCH`, and `nix-shell` sets this to 1. By setting it to a value corresponding to 1980 or later, or by unsetting it, it is possible to build wheels.
876
877Use 1980 as timestamp:
878```shell
879nix-shell --run "SOURCE_DATE_EPOCH=315532800 python3 setup.py bdist_wheel"
880```
881or the current time:
882```shell
883nix-shell --run "SOURCE_DATE_EPOCH=$(date +%s) python3 setup.py bdist_wheel"
884```
885or unset:
886```shell
887nix-shell --run "unset SOURCE_DATE_EPOCH; python3 setup.py bdist_wheel"
888```
889
890### `install_data` / `data_files` problems
891
892If you get the following error:
893```
894could not create '/nix/store/6l1bvljpy8gazlsw2aw9skwwp4pmvyxw-python-2.7.8/etc':
895Permission denied
896```
897This is a [known bug](https://github.com/pypa/setuptools/issues/130) in `setuptools`.
898Setuptools `install_data` does not respect `--prefix`. An example of such package using the feature is `pkgs/tools/X11/xpra/default.nix`.
899As workaround install it as an extra `preInstall` step:
900```shell
901${python.interpreter} setup.py install_data --install-dir=$out --root=$out
902sed -i '/ = data\_files/d' setup.py
903```
904
905### Rationale of non-existent global site-packages
906
907On most operating systems a global `site-packages` is maintained. This however
908becomes problematic if you want to run multiple Python versions or have multiple
909versions of certain libraries for your projects. Generally, you would solve such
910issues by creating virtual environments using `virtualenv`.
911
912On Nix each package has an isolated dependency tree which, in the case of
913Python, guarantees the right versions of the interpreter and libraries or
914packages are available. There is therefore no need to maintain a global `site-packages`.
915
916If you want to create a Python environment for development, then the recommended
917method is to use `nix-shell`, either with or without the `python.buildEnv`
918function.
919
920### How to consume python modules using pip in a virtualenv like I am used to on other Operating Systems ?
921
922This is an example of a `default.nix` for a `nix-shell`, which allows to consume a `virtualenv` environment,
923and install python modules through `pip` the traditional way.
924
925Create this `default.nix` file, together with a `requirements.txt` and simply execute `nix-shell`.
926
927```nix
928with import <nixpkgs> {};
929with pkgs.python27Packages;
930
931stdenv.mkDerivation {
932 name = "impurePythonEnv";
933 buildInputs = [
934 # these packages are required for virtualenv and pip to work:
935 #
936 python27Full
937 python27Packages.virtualenv
938 python27Packages.pip
939 # the following packages are related to the dependencies of your python
940 # project.
941 # In this particular example the python modules listed in the
942 # requirements.tx require the following packages to be installed locally
943 # in order to compile any binary extensions they may require.
944 #
945 taglib
946 openssl
947 git
948 libxml2
949 libxslt
950 libzip
951 stdenv
952 zlib ];
953 src = null;
954 shellHook = ''
955 # set SOURCE_DATE_EPOCH so that we can use python wheels
956 SOURCE_DATE_EPOCH=$(date +%s)
957 virtualenv --no-setuptools venv
958 export PATH=$PWD/venv/bin:$PATH
959 pip install -r requirements.txt
960 '';
961}
962```
963
964Note that the `pip install` is an imperative action. So every time `nix-shell`
965is executed it will attempt to download the python modules listed in
966requirements.txt. However these will be cached locally within the `virtualenv`
967folder and not downloaded again.
968
969### How to override a Python package from `configuration.nix`?
970
971If you need to change a package's attribute(s) from `configuration.nix` you could do:
972
973```nix
974 nixpkgs.config.packageOverrides = superP: {
975 pythonPackages = superP.pythonPackages.override {
976 overrides = self: super: {
977 bepasty-server = super.bepasty-server.overrideAttrs ( oldAttrs: {
978 src = pkgs.fetchgit {
979 url = "https://github.com/bepasty/bepasty-server";
980 sha256 = "9ziqshmsf0rjvdhhca55sm0x8jz76fsf2q4rwh4m6lpcf8wr0nps";
981 rev = "e2516e8cf4f2afb5185337073607eb9e84a61d2d";
982 };
983 });
984 };
985 };
986 };
987```
988
989If you are using the `bepasty-server` package somewhere, for example in `systemPackages` or indirectly from `services.bepasty`, then a `nixos-rebuild switch` will rebuild the system but with the `bepasty-server` package using a different `src` attribute. This way one can modify `python` based software/libraries easily. Using `self` and `super` one can also alter dependencies (`buildInputs`) between the old state (`self`) and new state (`super`).
990
991### How to override a Python package using overlays?
992
993To alter a python package using overlays, you would use the following approach:
994
995```nix
996self: super:
997rec {
998 python = super.python.override {
999 packageOverrides = python-self: python-super: {
1000 bepasty-server = python-super.bepasty-server.overrideAttrs ( oldAttrs: {
1001 src = self.pkgs.fetchgit {
1002 url = "https://github.com/bepasty/bepasty-server";
1003 sha256 = "9ziqshmsf0rjvdhhca55sm0x8jz76fsf2q4rwh4m6lpcf8wr0nps";
1004 rev = "e2516e8cf4f2afb5185337073607eb9e84a61d2d";
1005 };
1006 });
1007 };
1008 };
1009 pythonPackages = python.pkgs;
1010}
1011```
1012
1013## Contributing
1014
1015### Contributing guidelines
1016
1017Following rules are desired to be respected:
1018
1019* Python libraries are called from `python-packages.nix` and packaged with `buildPythonPackage`. The expression of a library should be in `pkgs/development/python-modules/<name>/default.nix`. Libraries in `pkgs/top-level/python-packages.nix` are sorted quasi-alphabetically to avoid merge conflicts.
1020* Python applications live outside of `python-packages.nix` and are packaged with `buildPythonApplication`.
1021* Make sure libraries build for all Python interpreters.
1022* By default we enable tests. Make sure the tests are found and, in the case of libraries, are passing for all interpreters. If certain tests fail they can be disabled individually. Try to avoid disabling the tests altogether. In any case, when you disable tests, leave a comment explaining why.
1023* Commit names of Python libraries should reflect that they are Python libraries, so write for example `pythonPackages.numpy: 1.11 -> 1.12`.
1024