···
5
-
Several versions of Python are available on Nix as well as a high amount of
6
-
packages. The default interpreter is CPython 2.7.
9
+
Several versions of the Python interpreter are available on Nix, as well as a
10
+
high amount of packages. The attribute `python` refers to the default
11
+
interpreter, which is currently CPython 2.7. It is also possible to refer to
12
+
specific versions, e.g. `python35` refers to CPython 3.5, and `pypy` refers to
13
+
the default PyPy interpreter.
15
+
Python is used a lot, and in different ways. This affects also how it is
16
+
packaged. In the case of Python on Nix, an important distinction is made between
17
+
whether the package is considered primarily an application, or whether it should
18
+
be used as a library, i.e., of primary interest are the modules in
19
+
`site-packages` that should be importable.
21
+
In the Nixpkgs tree Python applications can be found throughout, depending on
22
+
what they do, and are called from the main package set. Python libraries,
23
+
however, are in separate sets, with one set per interpreter version.
25
+
The interpreters have several common attributes. One of these attributes is
26
+
`pkgs`, which is a package set of Python libraries for this specific
27
+
interpreter. E.g., the `toolz` package corresponding to the default interpreter
28
+
is `python.pkgs.toolz`, and the CPython 3.5 version is `python35.pkgs.toolz`.
29
+
The main package set contains aliases to these package sets, e.g.
30
+
`pythonPackages` refers to `python.pkgs` and `python35Packages` to
#### Installing Python and packages
12
-
It is important to make a distinction between Python packages that are
13
-
used as libraries, and applications that are written in Python.
35
+
The Nix and NixOS manuals explain how packages are generally installed. In the
36
+
case of Python and Nix, it is important to make a distinction between whether the
37
+
package is considered an application or a library.
15
-
Applications on Nix are installed typically into your user
39
+
Applications on Nix are typically installed into your user
profile imperatively using `nix-env -i`, and on NixOS declaratively by adding the
package name to `environment.systemPackages` in `/etc/nixos/configuration.nix`.
Dependencies such as libraries are automatically installed and should not be
The same goes for Python applications and libraries. Python applications can be
22
-
installed in your profile, but Python libraries you would like to use to develop
23
-
cannot. If you do install libraries in your profile, then you will end up with
46
+
installed in your profile. But Python libraries you would like to use for
47
+
development cannot be installed, at least not individually, because they won't
48
+
be able to find each other resulting in import errors. Instead, it is possible
49
+
to create an environment with `python.buildEnv` or `python.withPackages` where
50
+
the interpreter and other executables are able to find each other and all of the
53
+
In the following examples we create an environment with Python 3.5, `numpy` and
54
+
`toolz`. As you may imagine, there is one limitation here, and that's that
55
+
you can install only one environment at a time. You will notice the complaints
56
+
about collisions when you try to install a second environment.
58
+
##### Environment defined in separate `.nix` file
60
+
Create a file, e.g. `build.nix`, with the following expression
62
+
with import <nixpkgs> {};
64
+
python35.withPackages (ps: with ps; [ numpy toolz ])
66
+
and install it in your profile with
68
+
nix-env -if build.nix
70
+
Now you can use the Python interpreter, as well as the extra packages (`numpy`,
71
+
`toolz`) that you added to the environment.
73
+
##### Environment defined in `~/.config/nixpkgs/config.nix`
75
+
If you prefer to, you could also add the environment as a package override to the Nixpkgs set, e.g.
80
+
packageOverrides = pkgs: with pkgs; {
81
+
myEnv = python35.withPackages (ps: with ps; [ numpy toolz ]);
85
+
and install it in your profile with
87
+
nix-env -iA nixpkgs.myEnv
89
+
The environment is is installed by referring to the attribute, and considering
90
+
the `nixpkgs` channel was used.
92
+
##### Environment defined in `/etc/nixos/configuration.nix`
26
-
#### Python environments using `nix-shell`
94
+
For the sake of completeness, here's another example how to install the environment system-wide.
28
-
The recommended method for creating Python environments for development is with
29
-
`nix-shell`. Executing
32
-
$ nix-shell -p python35Packages.numpy python35Packages.toolz
99
+
environment.systemPackages = with pkgs; [
100
+
(python35.withPackages(ps: with ps; [ numpy toolz ]))
35
-
opens a Nix shell which has available the requested packages and dependencies.
36
-
Now you can launch the Python interpreter (which is itself a dependency)
105
+
#### Temporary Python environment with `nix-shell`
107
+
The examples in the previous section showed how to install a Python environment
108
+
into a profile. For development you may need to use multiple environments.
109
+
`nix-shell` gives the possibility to temporarily load another environment, akin
112
+
There are two methods for loading a shell with Python packages. The first and recommended method
113
+
is to create an environment with `python.buildEnv` or `python.withPackages` and load that. E.g.
115
+
$ nix-shell -p 'python35.withPackages(ps: with ps; [ numpy toolz ])'
117
+
opens a shell from which you can launch the interpreter
121
+
The other method, which is not recommended, does not create an environment and requires you to list the packages directly,
42
-
If the packages were not available yet in the Nix store, Nix would download or
43
-
build them automatically. A convenient option with `nix-shell` is the `--run`
44
-
option, with which you can execute a command in the `nix-shell`. Let's say we
45
-
want the above environment and directly run the Python interpreter
124
+
$ nix-shell -p python35.pkgs.numpy python35.pkgs.toolz
126
+
Again, it is possible to launch the interpreter from the shell.
127
+
The Python interpreter has the attribute `pkgs` which contains all Python libraries for that specific interpreter.
129
+
##### Load environment from `.nix` expression
130
+
As explained in the Nix manual, `nix-shell` can also load an
131
+
expression from a `.nix` file. Say we want to have Python 3.5, `numpy`
132
+
and `toolz`, like before, in an environment. Consider a `shell.nix` file
135
+
with import <nixpkgs> {};
137
+
python35.withPackages (ps: [ps.numpy ps.toolz])
139
+
Executing `nix-shell` gives you again a Nix shell from which you can run Python.
141
+
What's happening here?
143
+
1. We begin with importing the Nix Packages collections. `import <nixpkgs>` imports the `<nixpkgs>` function, `{}` calls it and the `with` statement brings all attributes of `nixpkgs` in the local scope. These attributes form the main package set.
144
+
2. Then we create a Python 3.5 environment with the `withPackages` function.
145
+
3. The `withPackages` function expects us to provide a function as an argument that takes the set of all python packages and returns a list of packages to include in the environment. Here, we select the packages `numpy` and `toolz` from the package set.
147
+
##### Execute command with `--run`
148
+
A convenient option with `nix-shell` is the `--run`
149
+
option, with which you can execute a command in the `nix-shell`. We can
150
+
e.g. directly open a Python shell
$ nix-shell -p python35Packages.numpy python35Packages.toolz --run "python3"
51
-
This way you can use the `--run` option also to directly run a script
$ nix-shell -p python35Packages.numpy python35Packages.toolz --run "python3 myscript.py"
57
-
In fact, for this specific use case there is a more convenient method. You can
159
+
##### `nix-shell` as shebang
160
+
In fact, for the second use case, there is a more convenient method. You can
add a [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) to your script
59
-
specifying which dependencies Nix shell needs. With the following shebang, you
60
-
can use `nix-shell myscript.py` and it will make available all dependencies and
162
+
specifying which dependencies `nix-shell` needs. With the following shebang, you
163
+
can just execute `./myscript.py`, and it will make available all dependencies and
run the script in the `python3` shell.
#! /usr/bin/env nix-shell
65
-
#! nix-shell -i python3 -p python3Packages.numpy
168
+
#! nix-shell -i 'python3.withPackages(ps: [ps.numpy])'
72
-
Likely you do not want to type your dependencies each and every time. What you
73
-
can do is write a simple Nix expression which sets up an environment for you,
74
-
requiring you only to type `nix-shell`. Say we want to have Python 3.5, `numpy`
75
-
and `toolz`, like before, in an environment. With a `shell.nix` file
78
-
with import <nixpkgs> {};
80
-
(pkgs.python35.withPackages (ps: [ps.numpy ps.toolz])).env
82
-
executing `nix-shell` gives you again a Nix shell from which you can run Python.
84
-
What's happening here?
86
-
1. We begin with importing the Nix Packages collections. `import <nixpkgs>` import the `<nixpkgs>` function, `{}` calls it and the `with` statement brings all attributes of `nixpkgs` in the local scope. Therefore we can now use `pkgs`.
87
-
2. Then we create a Python 3.5 environment with the `withPackages` function.
88
-
3. The `withPackages` function expects us to provide a function as an argument that takes the set of all python packages and returns a list of packages to include in the environment. Here, we select the packages `numpy` and `toolz` from the package set.
89
-
4. And finally, for in interactive use we return the environment by using the `env` attribute.
### Developing with Python
177
+
Now that you know how to get a working Python environment with Nix, it is time
178
+
to go forward and start actually developing with Python. We will first have a
179
+
look at how Python packages are packaged on Nix. Then, we will look at how you
180
+
can use development mode with your code.
94
-
Now that you know how to get a working Python environment on Nix, it is time to go forward and start actually developing with Python.
95
-
We will first have a look at how Python packages are packaged on Nix. Then, we will look how you can use development mode with your code.
182
+
#### Packaging a library
97
-
#### Python packaging on Nix
99
-
On Nix all packages are built by functions. The main function in Nix for building Python packages is [`buildPythonPackage`](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/interpreters/python/build-python-package.nix).
100
-
Let's see how we would build the `toolz` package. According to [`python-packages.nix`](https://raw.githubusercontent.com/NixOS/nixpkgs/master/pkgs/top-level/python-packages.nix) `toolz` is build using
184
+
With Nix all packages are built by functions. The main function in Nix for
185
+
building Python libraries is `buildPythonPackage`. Let's see how we can build the
toolz = buildPythonPackage rec {
106
-
name = "toolz-${version}";
194
+
name = "${pname}-${version}";
109
-
src = pkgs.fetchurl {
110
-
url = "mirror://pypi/t/toolz/toolz-${version}.tar.gz";
197
+
inherit pname version;
sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed05afd";
homepage = "http://github.com/pytoolz/toolz/";
···
What happens here? The function `buildPythonPackage` is called and as argument
125
-
it accepts a set. In this case the set is a recursive set ([`rec`](http://nixos.org/nix/manual/#sec-constructs)).
126
-
One of the arguments is the name of the package, which consists of a basename
127
-
(generally following the name on PyPi) and a version. Another argument, `src`
128
-
specifies the source, which in this case is fetched from an url. `fetchurl` not
129
-
only downloads the target file, but also validates its hash. Furthermore, we
130
-
specify some (optional) [meta information](http://nixos.org/nixpkgs/manual/#chap-meta).
132
-
The output of the function is a derivation, which is an attribute with the name
133
-
`toolz` of the set `pythonPackages`. Actually, sets are created for all interpreter versions,
134
-
so e.g. `python27Packages`, `python35Packages` and `pypyPackages`.
214
+
it accepts a set. In this case the set is a recursive set, `rec`. One of the
215
+
arguments is the name of the package, which consists of a basename (generally
216
+
following the name on PyPi) and a version. Another argument, `src` specifies the
217
+
source, which in this case is fetched from PyPI using the helper function
218
+
`fetchPypi`. The argument `doCheck` is used to set whether tests should be run
219
+
when building the package. Furthermore, we specify some (optional) meta
220
+
information. The output of the function is a derivation.
222
+
An expression for `toolz` can be found in the Nixpkgs repository. As explained
223
+
in the introduction of this Python section, a derivation of `toolz` is available
224
+
for each interpreter version, e.g. `python35.pkgs.toolz` refers to the `toolz`
225
+
derivation corresponding to the CPython 3.5 interpreter.
The above example works when you're directly working on
`pkgs/top-level/python-packages.nix` in the Nixpkgs repository. Often though,
138
-
you will want to test a Nix expression outside of the Nixpkgs tree. If you
139
-
create a `shell.nix` file with the following contents
228
+
you will want to test a Nix expression outside of the Nixpkgs tree.
142
-
with import <nixpkgs> {};
144
-
pkgs.python35Packages.buildPythonPackage rec {
145
-
name = "toolz-${version}";
148
-
src = pkgs.fetchurl {
149
-
url = "mirror://pypi/t/toolz/toolz-${version}.tar.gz";
150
-
sha256 = "e8451af61face57b7c5d09e71c0d27b8005f001ead56e9fdf470417e5cc6d479";
156
-
homepage = "http://github.com/pytoolz/toolz/";
157
-
description = "List processing tools and functional utilities";
158
-
license = licenses.bsd3;
159
-
maintainers = with maintainers; [ fridh ];
164
-
and then execute `nix-shell` will result in an environment in which you can use
165
-
Python 3.5 and the `toolz` package. As you can see we had to explicitly mention
166
-
for which Python version we want to build a package.
168
-
The above example considered only a single package. Generally you will want to use multiple packages.
169
-
If we create a `shell.nix` file with the following contents
230
+
The following expression creates a derivation for the `toolz` package,
231
+
and adds it along with a `numpy` package to a Python environment.
with import <nixpkgs> {};
175
-
toolz = pkgs.python35Packages.buildPythonPackage rec {
176
-
name = "toolz-${version}";
237
+
my_toolz = python35.pkgs.buildPythonPackage rec {
240
+
name = "${pname}-${version}";
179
-
src = pkgs.fetchurl {
180
-
url = "mirror://pypi/t/toolz/toolz-${version}.tar.gz";
181
-
sha256 = "e8451af61face57b7c5d09e71c0d27b8005f001ead56e9fdf470417e5cc6d479";
242
+
src = python35.pkgs.fetchPypi {
243
+
inherit pname version;
244
+
sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed05afd";
···
192
-
in pkgs.python35.withPackages (ps: [ps.numpy toolz])
255
+
in python35.withPackages (ps: [ps.numpy my_toolz])
258
+
Executing `nix-shell` will result in an environment in which you can use
259
+
Python 3.5 and the `toolz` package. As you can see we had to explicitly mention
260
+
for which Python version we want to build a package.
196
-
and again execute `nix-shell`, then we get a Python 3.5 environment with our
197
-
locally defined package as well as `numpy` which is build according to the
198
-
definition in Nixpkgs. What did we do here? Well, we took the Nix expression
199
-
that we used earlier to build a Python environment, and said that we wanted to
200
-
include our own version of `toolz`. To introduce our own package in the scope of
201
-
`withPackages` we used a
202
-
[`let`](http://nixos.org/nix/manual/#sec-constructs) expression.
203
-
You can see that we used `ps.numpy` to select numpy from the nixpkgs package set (`ps`).
204
-
But we do not take `toolz` from the nixpkgs package set this time.
205
-
Instead, `toolz` will resolve to our local definition that we introduced with `let`.
262
+
So, what did we do here? Well, we took the Nix expression that we used earlier
263
+
to build a Python environment, and said that we wanted to include our own
264
+
version of `toolz`, named `my_toolz`. To introduce our own package in the scope
265
+
of `withPackages` we used a `let` expression. You can see that we used
266
+
`ps.numpy` to select numpy from the nixpkgs package set (`ps`). We did not take
267
+
`toolz` from the Nixpkgs package set this time, but instead took our own version
268
+
that we introduced with the `let` expression.
207
-
### Handling dependencies
270
+
#### Handling dependencies
209
-
Our example, `toolz`, doesn't have any dependencies on other Python
272
+
Our example, `toolz`, does not have any dependencies on other Python
packages or system libraries. According to the manual, `buildPythonPackage`
uses the arguments `buildInputs` and `propagatedBuildInputs` to specify dependencies. If something is
exclusively a build-time dependency, then the dependency should be included as a
···
716
-
### How can I install a working Python environment?
718
-
As explained in the user's guide installing individual Python packages
719
-
imperatively with `nix-env -i` or declaratively in `environment.systemPackages`
720
-
is not supported. However, it is possible to install a Python environment with packages (`python.buildEnv`).
722
-
In the following examples we create an environment with Python 3.5, `numpy` and `ipython`.
723
-
As you might imagine there is one limitation here, and that's you can install
724
-
only one environment at a time. You will notice the complaints about collisions
725
-
when you try to install a second environment.
727
-
#### Environment defined in separate `.nix` file
729
-
Create a file, e.g. `build.nix`, with the following expression
731
-
with import <nixpkgs> {};
733
-
pkgs.python35.withPackages (ps: with ps; [ numpy ipython ])
735
-
and install it in your profile with
737
-
nix-env -if build.nix
739
-
Now you can use the Python interpreter, as well as the extra packages that you added to the environment.
741
-
#### Environment defined in `~/.config/nixpkgs/config.nix`
743
-
If you prefer to, you could also add the environment as a package override to the Nixpkgs set.
747
-
packageOverrides = pkgs: with pkgs; {
748
-
myEnv = python35.withPackages (ps: with ps; [ numpy ipython ]);
752
-
and install it in your profile with
754
-
nix-env -iA nixpkgs.myEnv
757
-
We're installing using the attribute path and assume the channels is named `nixpkgs`.
758
-
Note that I'm using the attribute path here.
760
-
#### Environment defined in `/etc/nixos/configuration.nix`
762
-
For the sake of completeness, here's another example how to install the environment system-wide.
767
-
environment.systemPackages = with pkgs; [
768
-
(python35.withPackages(ps: with ps; [ numpy ipython ]))
### How to solve circular dependencies?
pyrox.dev