pyrox.dev
1# Python
2
3## User Guide
4
5Several versions of Python are available on Nix as well as a high amount of
6packages. The default interpreter is CPython 2.7.
7
8### Using Python
9
10#### Installing Python and packages
11
12It is important to make a distinction between Python packages that are
13used as libraries, and applications that are written in Python.
14
15Applications on Nix are installed typically into your user
16profile imperatively using `nix-env -i`, and on NixOS declaratively by adding the
17package name to `environment.systemPackages` in `/etc/nixos/configuration.nix`.
18Dependencies such as libraries are automatically installed and should not be
19installed explicitly.
20
21The same goes for Python applications and libraries. Python applications can be
22installed in your profile, but Python libraries you would like to use to develop
23cannot. If you do install libraries in your profile, then you will end up with
24import errors.
25
26#### Python environments using `nix-shell`
27
28The recommended method for creating Python environments for development is with
29`nix-shell`. Executing
30
31```sh
32$ nix-shell -p python35Packages.numpy python35Packages.toolz
33```
34
35opens a Nix shell which has available the requested packages and dependencies.
36Now you can launch the Python interpreter (which is itself a dependency)
37
38```sh
39[nix-shell:~] python3
40```
41
42If the packages were not available yet in the Nix store, Nix would download or
43build them automatically. A convenient option with `nix-shell` is the `--run`
44option, with which you can execute a command in the `nix-shell`. Let's say we
45want the above environment and directly run the Python interpreter
46
47```sh
48$ nix-shell -p python35Packages.numpy python35Packages.toolz --run "python3"
49```
50
51This way you can use the `--run` option also to directly run a script
52
53```sh
54$ nix-shell -p python35Packages.numpy python35Packages.toolz --run "python3 myscript.py"
55```
56
57In fact, for this specific use case there is a more convenient method. You can
58add a [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) to your script
59specifying which dependencies Nix shell needs. With the following shebang, you
60can use `nix-shell myscript.py` and it will make available all dependencies and
61run the script in the `python3` shell.
62
63```py
64#! /usr/bin/env nix-shell
65#! nix-shell -i python3 -p python3Packages.numpy
66
67import numpy
68
69print(numpy.__version__)
70```
71
72Likely you do not want to type your dependencies each and every time. What you
73can do is write a simple Nix expression which sets up an environment for you,
74requiring you only to type `nix-shell`. Say we want to have Python 3.5, `numpy`
75and `toolz`, like before, in an environment. With a `shell.nix` file
76containing
77
78```nix
79with import <nixpkgs> {};
80
81(pkgs.python35.withPackages (ps: [ps.numpy ps.toolz])).env
82```
83executing `nix-shell` gives you again a Nix shell from which you can run Python.
84
85What's happening here?
86
871. We begin with importing the Nix Packages collections. `import <nixpkgs>` import the `<nixpkgs>` function, `{}` calls it and the `with` statement brings all attributes of `nixpkgs` in the local scope. Therefore we can now use `pkgs`.
882. Then we create a Python 3.5 environment with the `withPackages` function.
893. The `withPackages` function expects us to provide a function as an argument that takes the set of all python packages and returns a list of packages to include in the environment. Here, we select the packages `numpy` and `toolz` from the package set.
904. And finally, for in interactive use we return the environment by using the `env` attribute.
91
92### Developing with Python
93
94
95Now that you know how to get a working Python environment on Nix, it is time to go forward and start actually developing with Python.
96We will first have a look at how Python packages are packaged on Nix. Then, we will look how you can use development mode with your code.
97
98#### Python packaging on Nix
99
100On Nix all packages are built by functions. The main function in Nix for building Python packages is [`buildPythonPackage`](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/python-modules/generic/default.nix).
101Let's see how we would build the `toolz` package. According to [`python-packages.nix`](https://raw.githubusercontent.com/NixOS/nixpkgs/master/pkgs/top-level/python-packages.nix) `toolz` is build using
102
103```nix
104toolz = buildPythonPackage rec{
105 name = "toolz-${version}";
106 version = "0.7.4";
107
108 src = pkgs.fetchurl{
109 url = "mirror://pypi/t/toolz/toolz-${version}.tar.gz";
110 sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed05afd";
111 };
112
113 meta = {
114 homepage = "http://github.com/pytoolz/toolz/";
115 description = "List processing tools and functional utilities";
116 license = licenses.bsd3;
117 maintainers = with maintainers; [ fridh ];
118 };
119};
120```
121
122What happens here? The function `buildPythonPackage` is called and as argument
123it accepts a set. In this case the set is a recursive set ([`rec`](http://nixos.org/nix/manual/#sec-constructs)).
124One of the arguments is the name of the package, which consists of a basename
125(generally following the name on PyPi) and a version. Another argument, `src`
126specifies the source, which in this case is fetched from an url. `fetchurl` not
127only downloads the target file, but also validates its hash. Furthermore, we
128specify some (optional) [meta information](http://nixos.org/nixpkgs/manual/#chap-meta).
129
130The output of the function is a derivation, which is an attribute with the name
131`toolz` of the set `pythonPackages`. Actually, sets are created for all interpreter versions,
132so `python27Packages`, `python34Packages`, `python35Packages` and `pypyPackages`.
133
134The above example works when you're directly working on
135`pkgs/top-level/python-packages.nix` in the Nixpkgs repository. Often though,
136you will want to test a Nix expression outside of the Nixpkgs tree. If you
137create a `shell.nix` file with the following contents
138
139```nix
140with import <nixpkgs> {};
141
142pkgs.python35Packages.buildPythonPackage rec {
143 name = "toolz-${version}";
144 version = "0.7.4";
145
146 src = pkgs.fetchurl{
147 url = "mirror://pypi/t/toolz/toolz-${version}.tar.gz";
148 sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed05afd";
149 };
150
151 meta = {
152 homepage = "http://github.com/pytoolz/toolz/";
153 description = "List processing tools and functional utilities";
154 license = licenses.bsd3;
155 maintainers = with maintainers; [ fridh ];
156 };
157}
158```
159
160and then execute `nix-shell` will result in an environment in which you can use
161Python 3.5 and the `toolz` package. As you can see we had to explicitly mention
162for which Python version we want to build a package.
163
164The above example considered only a single package. Generally you will want to use multiple packages.
165If we create a `shell.nix` file with the following contents
166
167```nix
168with import <nixpkgs> {};
169
170( let
171 toolz = pkgs.python35Packages.buildPythonPackage rec {
172 name = "toolz-${version}";
173 version = "0.7.4";
174
175 src = pkgs.fetchurl{
176 url = "mirror://pypi/t/toolz/toolz-${version}.tar.gz";
177 sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed05afd";
178 };
179
180 meta = {
181 homepage = "http://github.com/pytoolz/toolz/";
182 description = "List processing tools and functional utilities";
183 license = licenses.bsd3;
184 maintainers = with maintainers; [ fridh ];
185 };
186 };
187
188 in pkgs.python35.withPackages (ps: [ps.numpy toolz])
189).env
190```
191
192and again execute `nix-shell`, then we get a Python 3.5 environment with our
193locally defined package as well as `numpy` which is build according to the
194definition in Nixpkgs. What did we do here? Well, we took the Nix expression
195that we used earlier to build a Python environment, and said that we wanted to
196include our own version of `toolz`. To introduce our own package in the scope of
197`withPackages` we used a
198[`let`](http://nixos.org/nix/manual/#sec-constructs) expression.
199You can see that we used `ps.numpy` to select numpy from the nixpkgs package set (`ps`).
200But we do not take `toolz` from the nixpkgs package set this time.
201Instead, `toolz` will resolve to our local definition that we introduced with `let`.
202
203### Handling dependencies
204
205Our example, `toolz`, doesn't have any dependencies on other Python
206packages or system libraries. According to the manual, `buildPythonPackage`
207uses the arguments `buildInputs` and `propagatedBuildInputs` to specify dependencies. If something is
208exclusively a build-time dependency, then the dependency should be included as a
209`buildInput`, but if it is (also) a runtime dependency, then it should be added
210to `propagatedBuildInputs`. Test dependencies are considered build-time dependencies.
211
212The following example shows which arguments are given to `buildPythonPackage` in
213order to build [`datashape`](https://github.com/blaze/datashape).
214
215```nix
216datashape = buildPythonPackage rec {
217 name = "datashape-${version}";
218 version = "0.4.7";
219
220 src = pkgs.fetchurl {
221 url = "mirror://pypi/D/DataShape/${name}.tar.gz";
222 sha256 = "14b2ef766d4c9652ab813182e866f493475e65e558bed0822e38bf07bba1a278";
223 };
224
225 buildInputs = with self; [ pytest ];
226 propagatedBuildInputs = with self; [ numpy multipledispatch dateutil ];
227
228 meta = {
229 homepage = https://github.com/ContinuumIO/datashape;
230 description = "A data description language";
231 license = licenses.bsd2;
232 maintainers = with maintainers; [ fridh ];
233 };
234};
235```
236
237We can see several runtime dependencies, `numpy`, `multipledispatch`, and
238`dateutil`. Furthermore, we have one `buildInput`, i.e. `pytest`. `pytest` is a
239test runner and is only used during the `checkPhase` and is therefore not added
240to `propagatedBuildInputs`.
241
242In the previous case we had only dependencies on other Python packages to consider.
243Occasionally you have also system libraries to consider. E.g., `lxml` provides
244Python bindings to `libxml2` and `libxslt`. These libraries are only required
245when building the bindings and are therefore added as `buildInputs`.
246
247```nix
248lxml = buildPythonPackage rec {
249 name = "lxml-3.4.4";
250
251 src = pkgs.fetchurl {
252 url = "mirror://pypi/l/lxml/${name}.tar.gz";
253 sha256 = "16a0fa97hym9ysdk3rmqz32xdjqmy4w34ld3rm3jf5viqjx65lxk";
254 };
255
256 buildInputs = with self; [ pkgs.libxml2 pkgs.libxslt ];
257
258 meta = {
259 description = "Pythonic binding for the libxml2 and libxslt libraries";
260 homepage = http://lxml.de;
261 license = licenses.bsd3;
262 maintainers = with maintainers; [ sjourdois ];
263 };
264};
265```
266
267In this example `lxml` and Nix are able to work out exactly where the relevant
268files of the dependencies are. This is not always the case.
269
270The example below shows bindings to The Fastest Fourier Transform in the West, commonly known as
271FFTW. On Nix we have separate packages of FFTW for the different types of floats
272(`"single"`, `"double"`, `"long-double"`). The bindings need all three types,
273and therefore we add all three as `buildInputs`. The bindings don't expect to
274find each of them in a different folder, and therefore we have to set `LDFLAGS`
275and `CFLAGS`.
276
277```nix
278pyfftw = buildPythonPackage rec {
279 name = "pyfftw-${version}";
280 version = "0.9.2";
281
282 src = pkgs.fetchurl {
283 url = "mirror://pypi/p/pyFFTW/pyFFTW-${version}.tar.gz";
284 sha256 = "f6bbb6afa93085409ab24885a1a3cdb8909f095a142f4d49e346f2bd1b789074";
285 };
286
287 buildInputs = [ pkgs.fftw pkgs.fftwFloat pkgs.fftwLongDouble];
288
289 propagatedBuildInputs = with self; [ numpy scipy ];
290
291 # Tests cannot import pyfftw. pyfftw works fine though.
292 doCheck = false;
293
294 LDFLAGS="-L${pkgs.fftw.dev}/lib -L${pkgs.fftwFloat.out}/lib -L${pkgs.fftwLongDouble.out}/lib"
295 CFLAGS="-I${pkgs.fftw.dev}/include -I${pkgs.fftwFloat.dev}/include -I${pkgs.fftwLongDouble.dev}/include"
296 '';
297
298 meta = {
299 description = "A pythonic wrapper around FFTW, the FFT library, presenting a unified interface for all the supported transforms";
300 homepage = http://hgomersall.github.com/pyFFTW/;
301 license = with licenses; [ bsd2 bsd3 ];
302 maintainer = with maintainers; [ fridh ];
303 };
304};
305```
306Note also the line `doCheck = false;`, we explicitly disabled running the test-suite.
307
308
309#### Develop local package
310
311As a Python developer you're likely aware of [development mode](http://pythonhosted.org/setuptools/setuptools.html#development-mode) (`python setup.py develop`);
312instead of installing the package this command creates a special link to the project code.
313That way, you can run updated code without having to reinstall after each and every change you make.
314Development mode is also available on Nix as [explained](http://nixos.org/nixpkgs/manual/#ssec-python-development) in the Nixpkgs manual.
315Let's see how you can use it.
316
317In the previous Nix expression the source was fetched from an url. We can also refer to a local source instead using
318
319```nix
320src = ./path/to/source/tree;
321```
322
323If we create a `shell.nix` file which calls `buildPythonPackage`, and if `src`
324is a local source, and if the local source has a `setup.py`, then development
325mode is activated.
326
327In the following example we create a simple environment that
328has a Python 3.5 version of our package in it, as well as its dependencies and
329other packages we like to have in the environment, all specified with `propagatedBuildInputs`.
330Indeed, we can just add any package we like to have in our environment to `propagatedBuildInputs`.
331
332```nix
333with import <nixpkgs>;
334with pkgs.python35Packages;
335
336buildPythonPackage rec {
337 name = "mypackage";
338 src = ./path/to/package/source;
339 propagatedBuildInputs = [ pytest numpy pkgs.libsndfile ];
340};
341```
342
343It is important to note that due to how development mode is implemented on Nix it is not possible to have multiple packages simultaneously in development mode.
344
345
346### Organising your packages
347
348So far we discussed how you can use Python on Nix, and how you can develop with
349it. We've looked at how you write expressions to package Python packages, and we
350looked at how you can create environments in which specified packages are
351available.
352
353At some point you'll likely have multiple packages which you would
354like to be able to use in different projects. In order to minimise unnecessary
355duplication we now look at how you can maintain yourself a repository with your
356own packages. The important functions here are `import` and `callPackage`.
357
358### Including a derivation using `callPackage`
359
360Earlier we created a Python environment using `withPackages`, and included the
361`toolz` package via a `let` expression.
362Let's split the package definition from the environment definition.
363
364We first create a function that builds `toolz` in `~/path/to/toolz/release.nix`
365
366```nix
367{ pkgs, buildPythonPackage }:
368
369buildPythonPackage rec {
370 name = "toolz-${version}";
371 version = "0.7.4";
372
373 src = pkgs.fetchurl{
374 url = "mirror://pypi/t/toolz/toolz-${version}.tar.gz";
375 sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed05afd";
376 };
377
378 meta = {
379 homepage = "http://github.com/pytoolz/toolz/";
380 description = "List processing tools and functional utilities";
381 license = licenses.bsd3;
382 maintainers = with maintainers; [ fridh ];
383 };
384};
385```
386
387It takes two arguments, `pkgs` and `buildPythonPackage`.
388We now call this function using `callPackage` in the definition of our environment
389
390```nix
391with import <nixpkgs> {};
392
393( let
394 toolz = pkgs.callPackage ~/path/to/toolz/release.nix { pkgs=pkgs; buildPythonPackage=pkgs.python35Packages.buildPythonPackage; };
395 in pkgs.python35.withPackages (ps: [ ps.numpy toolz ])
396).env
397```
398
399Important to remember is that the Python version for which the package is made
400depends on the `python` derivation that is passed to `buildPythonPackage`. Nix
401tries to automatically pass arguments when possible, which is why generally you
402don't explicitly define which `python` derivation should be used. In the above
403example we use `buildPythonPackage` that is part of the set `python35Packages`,
404and in this case the `python35` interpreter is automatically used.
405
406
407
408## Reference
409
410### Interpreters
411
412Versions 2.6, 2.7, 3.3, 3.4 and 3.5 of the CPython interpreter are available on
413Nix and are available as `python26`, `python27`, `python33`, `python34` and
414`python35`. The PyPy interpreter is also available as `pypy`. Currently, the
415aliases `python` and `python3` correspond to respectively `python27` and
416`python35`. The Nix expressions for the interpreters can be found in
417`pkgs/development/interpreters/python`.
418
419
420#### Missing modules standard library
421
422The interpreters `python26` and `python27` do not include modules that
423require external dependencies. This is done in order to reduce the closure size.
424The following modules need to be added as `buildInput` explicitly:
425
426* `python.modules.bsddb`
427* `python.modules.curses`
428* `python.modules.curses_panel`
429* `python.modules.crypt`
430* `python.modules.gdbm`
431* `python.modules.sqlite3`
432* `python.modules.tkinter`
433* `python.modules.readline`
434
435For convenience `python27Full` and `python26Full` are provided with all
436modules included.
437
438All packages depending on any Python interpreter get appended
439`out/{python.sitePackages}` to `$PYTHONPATH` if such directory
440exists.
441
442#### Attributes on interpreters packages
443
444Each interpreter has the following attributes:
445
446- `libPrefix`. Name of the folder in `${python}/lib/` for corresponding interpreter.
447- `interpreter`. Alias for `${python}/bin/${executable}`.
448- `buildEnv`. Function to build python interpreter environments with extra packages bundled together. See section *python.buildEnv function* for usage and documentation.
449- `withPackages`. Simpler interface to `buildEnv`. See section *python.withPackages function* for usage and documentation.
450- `sitePackages`. Alias for `lib/${libPrefix}/site-packages`.
451- `executable`. Name of the interpreter executable, ie `python3.4`.
452
453### Building packages and applications
454
455Python packages (libraries) and applications that use `setuptools` or
456`distutils` are typically built with respectively the `buildPythonPackage` and
457`buildPythonApplication` functions.
458
459All Python packages reside in `pkgs/top-level/python-packages.nix` and all
460applications elsewhere. Some packages are also defined in
461`pkgs/development/python-modules`. It is important that these packages are
462called in `pkgs/top-level/python-packages.nix` and not elsewhere, to guarantee
463the right version of the package is built.
464
465Based on the packages defined in `pkgs/top-level/python-packages.nix` an
466attribute set is created for each available Python interpreter. The available
467sets are
468
469* `pkgs.python26Packages`
470* `pkgs.python27Packages`
471* `pkgs.python33Packages`
472* `pkgs.python34Packages`
473* `pkgs.python35Packages`
474* `pkgs.pypyPackages`
475
476and the aliases
477
478* `pkgs.pythonPackages` pointing to `pkgs.python27Packages`
479* `pkgs.python3Packages` pointing to `pkgs.python35Packages`
480
481#### `buildPythonPackage` function
482
483The `buildPythonPackage` function is implemented in
484`pkgs/development/interpreters/python/build-python-package.nix`
485
486and can be used as:
487
488 twisted = buildPythonPackage {
489 name = "twisted-8.1.0";
490
491 src = pkgs.fetchurl {
492 url = http://tmrc.mit.edu/mirror/twisted/Twisted/8.1/Twisted-8.1.0.tar.bz2;
493 sha256 = "0q25zbr4xzknaghha72mq57kh53qw1bf8csgp63pm9sfi72qhirl";
494 };
495
496 propagatedBuildInputs = [ self.ZopeInterface ];
497
498 meta = {
499 homepage = http://twistedmatrix.com/;
500 description = "Twisted, an event-driven networking engine written in Python";
501 license = stdenv.lib.licenses.mit; };
502 };
503
504The `buildPythonPackage` mainly does four things:
505
506* In the `buildPhase`, it calls `${python.interpreter} setup.py bdist_wheel` to
507 build a wheel binary zipfile.
508* In the `installPhase`, it installs the wheel file using `pip install *.whl`.
509* In the `postFixup` phase, the `wrapPythonPrograms` bash function is called to
510 wrap all programs in the `$out/bin/*` directory to include `$PATH`
511 environment variable and add dependent libraries to script's `sys.path`.
512* In the `installCheck` phase, `${python.interpreter} setup.py test` is ran.
513
514As in Perl, dependencies on other Python packages can be specified in the
515`buildInputs` and `propagatedBuildInputs` attributes. If something is
516exclusively a build-time dependency, use `buildInputs`; if it’s (also) a runtime
517dependency, use `propagatedBuildInputs`.
518
519By default tests are run because `doCheck = true`. Test dependencies, like
520e.g. the test runner, should be added to `buildInputs`.
521
522By default `meta.platforms` is set to the same value
523as the interpreter unless overriden otherwise.
524
525##### `buildPythonPackage` parameters
526
527All parameters from `mkDerivation` function are still supported.
528
529* `namePrefix`: Prepended text to `${name}` parameter. Defaults to `"python3.3-"` for Python 3.3, etc. Set it to `""` if you're packaging an application or a command line tool.
530* `disabled`: If `true`, package is not build for particular python interpreter version. Grep around `pkgs/top-level/python-packages.nix` for examples.
531* `setupPyBuildFlags`: List of flags passed to `setup.py build_ext` command.
532* `pythonPath`: List of packages to be added into `$PYTHONPATH`. Packages in `pythonPath` are not propagated (contrary to `propagatedBuildInputs`).
533* `preShellHook`: Hook to execute commands before `shellHook`.
534* `postShellHook`: Hook to execute commands after `shellHook`.
535* `makeWrapperArgs`: A list of strings. Arguments to be passed to `makeWrapper`, which wraps generated binaries. By default, the arguments to `makeWrapper` set `PATH` and `PYTHONPATH` environment variables before calling the binary. Additional arguments here can allow a developer to set environment variables which will be available when the binary is run. For example, `makeWrapperArgs = ["--set FOO BAR" "--set BAZ QUX"]`.
536* `installFlags`: A list of strings. Arguments to be passed to `pip install`. To pass options to `python setup.py install`, use `--install-option`. E.g., `installFlags=["--install-option='--cpp_implementation'"].
537* `format`: Format of the source. Options are `setup` for when the source has a `setup.py` and `setuptools` is used to build a wheel, and `wheel` in case the source is already a binary wheel. The default value is `setup`.
538* `catchConflicts` If `true`, abort package build if a package name appears more than once in dependency tree. Default is `true`.
539* `checkInputs` Dependencies needed for running the `checkPhase`. These are added to `buildInputs` when `doCheck = true`.
540
541#### `buildPythonApplication` function
542
543The `buildPythonApplication` function is practically the same as `buildPythonPackage`.
544The difference is that `buildPythonPackage` by default prefixes the names of the packages with the version of the interpreter.
545Because with an application we're not interested in multiple version the prefix is dropped.
546
547#### python.buildEnv function
548
549Python environments can be created using the low-level `pkgs.buildEnv` function.
550This example shows how to create an environment that has the Pyramid Web Framework.
551Saving the following as `default.nix`
552
553 with import <nixpkgs> {};
554
555 python.buildEnv.override {
556 extraLibs = [ pkgs.pythonPackages.pyramid ];
557 ignoreCollisions = true;
558 }
559
560and running `nix-build` will create
561
562 /nix/store/cf1xhjwzmdki7fasgr4kz6di72ykicl5-python-2.7.8-env
563
564with wrapped binaries in `bin/`.
565
566You can also use the `env` attribute to create local environments with needed
567packages installed. This is somewhat comparable to `virtualenv`. For example,
568running `nix-shell` with the following `shell.nix`
569
570 with import <nixpkgs> {};
571
572 (python3.buildEnv.override {
573 extraLibs = with python3Packages; [ numpy requests2 ];
574 }).env
575
576will drop you into a shell where Python will have the
577specified packages in its path.
578
579
580##### `python.buildEnv` arguments
581
582* `extraLibs`: List of packages installed inside the environment.
583* `postBuild`: Shell command executed after the build of environment.
584* `ignoreCollisions`: Ignore file collisions inside the environment (default is `false`).
585
586#### python.withPackages function
587
588The `python.withPackages` function provides a simpler interface to the `python.buildEnv` functionality.
589It takes a function as an argument that is passed the set of python packages and returns the list
590of the packages to be included in the environment. Using the `withPackages` function, the previous
591example for the Pyramid Web Framework environment can be written like this:
592
593 with import <nixpkgs> {};
594
595 python.withPackages (ps: [ps.pyramid])
596
597`withPackages` passes the correct package set for the specific interpreter version as an
598argument to the function. In the above example, `ps` equals `pythonPackages`.
599But you can also easily switch to using python3:
600
601 with import <nixpkgs> {};
602
603 python3.withPackages (ps: [ps.pyramid])
604
605Now, `ps` is set to `python3Packages`, matching the version of the interpreter.
606
607As `python.withPackages` simply uses `python.buildEnv` under the hood, it also supports the `env`
608attribute. The `shell.nix` file from the previous section can thus be also written like this:
609
610 with import <nixpkgs> {};
611
612 (python33.withPackages (ps: [ps.numpy ps.requests2])).env
613
614In contrast to `python.buildEnv`, `python.withPackages` does not support the more advanced options
615such as `ignoreCollisions = true` or `postBuild`. If you need them, you have to use `python.buildEnv`.
616
617### Development mode
618
619Development or editable mode is supported. To develop Python packages
620`buildPythonPackage` has additional logic inside `shellPhase` to run `pip
621install -e . --prefix $TMPDIR/`for the package.
622
623Warning: `shellPhase` is executed only if `setup.py` exists.
624
625Given a `default.nix`:
626
627 with import <nixpkgs> {};
628
629 buildPythonPackage { name = "myproject";
630
631 buildInputs = with pkgs.pythonPackages; [ pyramid ];
632
633 src = ./.; }
634
635Running `nix-shell` with no arguments should give you
636the environment in which the package would be built with
637`nix-build`.
638
639Shortcut to setup environments with C headers/libraries and python packages:
640
641 $ nix-shell -p pythonPackages.pyramid zlib libjpeg git
642
643Note: There is a boolean value `lib.inNixShell` set to `true` if nix-shell is invoked.
644
645### Tools
646
647Packages inside nixpkgs are written by hand. However many tools exist in
648community to help save time. No tool is preferred at the moment.
649
650- [python2nix](https://github.com/proger/python2nix) by Vladimir Kirillov
651- [pypi2nix](https://github.com/garbas/pypi2nix) by Rok Garbas
652- [pypi2nix](https://github.com/offlinehacker/pypi2nix) by Jaka Hudoklin
653
654## FAQ
655
656### How can I install a working Python environment?
657
658As explained in the user's guide installing individual Python packages
659imperatively with `nix-env -i` or declaratively in `environment.systemPackages`
660is not supported. However, it is possible to install a Python environment with packages (`python.buildEnv`).
661
662In the following examples we create an environment with Python 3.5, `numpy` and `ipython`.
663As you might imagine there is one limitation here, and that's you can install
664only one environment at a time. You will notice the complaints about collisions
665when you try to install a second environment.
666
667#### Environment defined in separate `.nix` file
668
669Create a file, e.g. `build.nix`, with the following expression
670```nix
671with import <nixpkgs> {};
672with python35Packages;
673
674python.withPackages (ps: with ps; [ numpy ipython ])
675```
676and install it in your profile with
677```
678nix-env -if build.nix
679```
680Now you can use the Python interpreter, as well as the extra packages that you added to the environment.
681
682#### Environment defined in `~/.nixpkgs/config.nix`
683
684If you prefer to, you could also add the environment as a package override to the Nixpkgs set.
685```
686 packageOverrides = pkgs: with pkgs; with python35Packages; {
687 myEnv = python.withPackages (ps: with ps; [ numpy ipython ]);
688 };
689```
690and install it in your profile with
691```
692nix-env -iA nixos.blogEnv
693```
694Note that I'm using the attribute path here.
695
696#### Environment defined in `/etc/nixos/configuration.nix`
697
698For the sake of completeness, here's another example how to install the environment system-wide.
699
700```nix
701environment.systemPackages = with pkgs; [
702 (python35Packages.python.withPackages (ps: callPackage ../packages/common-python-packages.nix { pythonPackages = ps; }))
703];
704```
705
706### How to solve circular dependencies?
707
708Consider the packages `A` and `B` that depend on each other. When packaging `B`,
709a solution is to override package `A` not to depend on `B` as an input. The same
710should also be done when packaging `A`.
711
712### How to override a Python package?
713
714Recursively updating a package can be done with `pkgs.overridePackages` as explained in the Nixpkgs manual.
715Python attribute sets are created for each interpreter version. We will therefore override the attribute set for the interpreter version we're interested.
716In the following example we change the name of the package `pandas` to `foo`.
717```
718newpkgs = pkgs.overridePackages(self: super: rec {
719 python35Packages = (super.python35Packages.override { self = python35Packages;})
720 // { pandas = super.python35Packages.pandas.override {name = "foo";};
721 };
722});
723```
724This can be tested with
725```
726with import <nixpkgs> {};
727
728(let
729
730newpkgs = pkgs.overridePackages(self: super: rec {
731 python35Packages = (super.python35Packages.override { self = python35Packages;})
732 // { pandas = super.python35Packages.pandas.override {name = "foo";};
733 };
734});
735in newpkgs.python35.withPackages (ps: [ps.blaze])
736).env
737```
738A typical use case is to switch to another version of a certain package. For example, in the Nixpkgs repository we have multiple versions of `django` and `scipy`.
739In the following example we use a different version of `scipy`. All packages in `newpkgs` will now use the updated `scipy` version.
740```
741with import <nixpkgs> {};
742
743(let
744
745newpkgs = pkgs.overridePackages(self: super: rec {
746 python35Packages = super.python35Packages.override {
747 self = python35Packages // { scipy = python35Packages.scipy_0_17;};
748 };
749});
750in newpkgs.python35.withPackages (ps: [ps.blaze])
751).env
752```
753The requested package `blaze` depends upon `pandas` which itself depends on `scipy`.
754
755A similar example but now using `django`
756```
757with import <nixpkgs> {};
758
759(let
760
761newpkgs = pkgs.overridePackages(self: super: rec {
762 python27Packages = (super.python27Packages.override {self = python27Packages;})
763 // { django = super.python27Packages.django_1_9; };
764});
765in newpkgs.python27.withPackages (ps: [ps.django_guardian ])
766).env
767```
768
769### `python setup.py bdist_wheel` cannot create .whl
770
771Executing `python setup.py bdist_wheel` in a `nix-shell `fails with
772```
773ValueError: ZIP does not support timestamps before 1980
774```
775This is because files are included that depend on items in the Nix store which have a timestamp of, that is, it corresponds to January the 1st, 1970 at 00:00:00. And as the error informs you, ZIP does not support that.
776The command `bdist_wheel` takes into account `SOURCE_DATE_EPOCH`, and `nix-shell` sets this to 1. By setting it to a value corresponding to 1980 or later, or by unsetting it, it is possible to build wheels.
777
778Use 1980 as timestamp:
779```
780nix-shell --run "SOURCE_DATE_EPOCH=315532800 python3 setup.py bdist_wheel"
781```
782or the current time:
783```
784nix-shell --run "SOURCE_DATE_EPOCH=$(date +%s) python3 setup.py bdist_wheel"
785```
786or unset:
787"""
788nix-shell --run "unset SOURCE_DATE_EPOCH; python3 setup.py bdist_wheel"
789"""
790
791### `install_data` / `data_files` problems
792
793If you get the following error:
794
795 could not create '/nix/store/6l1bvljpy8gazlsw2aw9skwwp4pmvyxw-python-2.7.8/etc':
796 Permission denied
797
798This is a [known bug](https://bitbucket.org/pypa/setuptools/issue/130/install_data-doesnt-respect-prefix) in setuptools.
799Setuptools `install_data` does not respect `--prefix`. An example of such package using the feature is `pkgs/tools/X11/xpra/default.nix`.
800As workaround install it as an extra `preInstall` step:
801
802 ${python.interpreter} setup.py install_data --install-dir=$out --root=$out
803 sed -i '/ = data\_files/d' setup.py
804
805
806### Rationale of non-existent global site-packages
807
808On most operating systems a global `site-packages` is maintained. This however
809becomes problematic if you want to run multiple Python versions or have multiple
810versions of certain libraries for your projects. Generally, you would solve such
811issues by creating virtual environments using `virtualenv`.
812
813On Nix each package has an isolated dependency tree which, in the case of
814Python, guarantees the right versions of the interpreter and libraries or
815packages are available. There is therefore no need to maintain a global `site-packages`.
816
817If you want to create a Python environment for development, then the recommended
818method is to use `nix-shell`, either with or without the `python.buildEnv`
819function.
820
821
822## Contributing
823
824### Contributing guidelines
825
826Following rules are desired to be respected:
827
828* Make sure package builds for all python interpreters. Use `disabled` argument to `buildPythonPackage` to set unsupported interpreters.
829* If tests need to be disabled for a package, make sure you leave a comment about reasoning.
830* Packages in `pkgs/top-level/python-packages.nix` are sorted quasi-alphabetically to avoid merge conflicts.
831* Python libraries are supposed to be in `python-packages.nix` and packaged with `buildPythonPackage`. Python applications live outside of `python-packages.nix` and are packaged with `buildPythonApplication`.