···
+
Several versions of the Python interpreter are available on Nix, as well as a
+
high amount of packages. The attribute `python` refers to the default
+
interpreter, which is currently CPython 2.7. It is also possible to refer to
+
specific versions, e.g. `python35` refers to CPython 3.5, and `pypy` refers to
+
the default PyPy interpreter.
+
Python is used a lot, and in different ways. This affects also how it is
+
packaged. In the case of Python on Nix, an important distinction is made between
+
whether the package is considered primarily an application, or whether it should
+
be used as a library, i.e., of primary interest are the modules in
+
`site-packages` that should be importable.
+
In the Nixpkgs tree Python applications can be found throughout, depending on
+
what they do, and are called from the main package set. Python libraries,
+
however, are in separate sets, with one set per interpreter version.
+
The interpreters have several common attributes. One of these attributes is
+
`pkgs`, which is a package set of Python libraries for this specific
+
interpreter. E.g., the `toolz` package corresponding to the default interpreter
+
is `python.pkgs.toolz`, and the CPython 3.5 version is `python35.pkgs.toolz`.
+
The main package set contains aliases to these package sets, e.g.
+
`pythonPackages` refers to `python.pkgs` and `python35Packages` to
#### Installing Python and packages
+
The Nix and NixOS manuals explain how packages are generally installed. In the
+
case of Python and Nix, it is important to make a distinction between whether the
+
package is considered an application or a library.
+
Applications on Nix are typically installed into your user
profile imperatively using `nix-env -i`, and on NixOS declaratively by adding the
package name to `environment.systemPackages` in `/etc/nixos/configuration.nix`.
Dependencies such as libraries are automatically installed and should not be
The same goes for Python applications and libraries. Python applications can be
+
installed in your profile. But Python libraries you would like to use for
+
development cannot be installed, at least not individually, because they won't
+
be able to find each other resulting in import errors. Instead, it is possible
+
to create an environment with `python.buildEnv` or `python.withPackages` where
+
the interpreter and other executables are able to find each other and all of the
+
In the following examples we create an environment with Python 3.5, `numpy` and
+
`toolz`. As you may imagine, there is one limitation here, and that's that
+
you can install only one environment at a time. You will notice the complaints
+
about collisions when you try to install a second environment.
+
##### Environment defined in separate `.nix` file
+
Create a file, e.g. `build.nix`, with the following expression
+
with import <nixpkgs> {};
+
python35.withPackages (ps: with ps; [ numpy toolz ])
+
and install it in your profile with
+
Now you can use the Python interpreter, as well as the extra packages (`numpy`,
+
`toolz`) that you added to the environment.
+
##### Environment defined in `~/.config/nixpkgs/config.nix`
+
If you prefer to, you could also add the environment as a package override to the Nixpkgs set, e.g.
+
packageOverrides = pkgs: with pkgs; {
+
myEnv = python35.withPackages (ps: with ps; [ numpy toolz ]);
+
and install it in your profile with
+
nix-env -iA nixpkgs.myEnv
+
The environment is is installed by referring to the attribute, and considering
+
the `nixpkgs` channel was used.
+
##### Environment defined in `/etc/nixos/configuration.nix`
+
For the sake of completeness, here's another example how to install the environment system-wide.
+
environment.systemPackages = with pkgs; [
+
(python35.withPackages(ps: with ps; [ numpy toolz ]))
+
#### Temporary Python environment with `nix-shell`
+
The examples in the previous section showed how to install a Python environment
+
into a profile. For development you may need to use multiple environments.
+
`nix-shell` gives the possibility to temporarily load another environment, akin
+
There are two methods for loading a shell with Python packages. The first and recommended method
+
is to create an environment with `python.buildEnv` or `python.withPackages` and load that. E.g.
+
$ nix-shell -p 'python35.withPackages(ps: with ps; [ numpy toolz ])'
+
opens a shell from which you can launch the interpreter
+
The other method, which is not recommended, does not create an environment and requires you to list the packages directly,
+
$ nix-shell -p python35.pkgs.numpy python35.pkgs.toolz
+
Again, it is possible to launch the interpreter from the shell.
+
The Python interpreter has the attribute `pkgs` which contains all Python libraries for that specific interpreter.
+
##### Load environment from `.nix` expression
+
As explained in the Nix manual, `nix-shell` can also load an
+
expression from a `.nix` file. Say we want to have Python 3.5, `numpy`
+
and `toolz`, like before, in an environment. Consider a `shell.nix` file
+
with import <nixpkgs> {};
+
python35.withPackages (ps: [ps.numpy ps.toolz])
+
Executing `nix-shell` gives you again a Nix shell from which you can run Python.
+
1. We begin with importing the Nix Packages collections. `import <nixpkgs>` imports the `<nixpkgs>` function, `{}` calls it and the `with` statement brings all attributes of `nixpkgs` in the local scope. These attributes form the main package set.
+
2. Then we create a Python 3.5 environment with the `withPackages` function.
+
3. The `withPackages` function expects us to provide a function as an argument that takes the set of all python packages and returns a list of packages to include in the environment. Here, we select the packages `numpy` and `toolz` from the package set.
+
##### Execute command with `--run`
+
A convenient option with `nix-shell` is the `--run`
+
option, with which you can execute a command in the `nix-shell`. We can
+
e.g. directly open a Python shell
$ nix-shell -p python35Packages.numpy python35Packages.toolz --run "python3"
$ nix-shell -p python35Packages.numpy python35Packages.toolz --run "python3 myscript.py"
+
##### `nix-shell` as shebang
+
In fact, for the second use case, there is a more convenient method. You can
add a [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) to your script
+
specifying which dependencies `nix-shell` needs. With the following shebang, you
+
can just execute `./myscript.py`, and it will make available all dependencies and
run the script in the `python3` shell.
#! /usr/bin/env nix-shell
+
#! nix-shell -i 'python3.withPackages(ps: [ps.numpy])'
### Developing with Python
+
Now that you know how to get a working Python environment with Nix, it is time
+
to go forward and start actually developing with Python. We will first have a
+
look at how Python packages are packaged on Nix. Then, we will look at how you
+
can use development mode with your code.
+
#### Packaging a library
+
With Nix all packages are built by functions. The main function in Nix for
+
building Python libraries is `buildPythonPackage`. Let's see how we can build the
toolz = buildPythonPackage rec {
+
name = "${pname}-${version}";
sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed05afd";
homepage = "http://github.com/pytoolz/toolz/";
···
What happens here? The function `buildPythonPackage` is called and as argument
+
it accepts a set. In this case the set is a recursive set, `rec`. One of the
+
arguments is the name of the package, which consists of a basename (generally
+
following the name on PyPi) and a version. Another argument, `src` specifies the
+
source, which in this case is fetched from PyPI using the helper function
+
`fetchPypi`. The argument `doCheck` is used to set whether tests should be run
+
when building the package. Furthermore, we specify some (optional) meta
+
information. The output of the function is a derivation.
+
An expression for `toolz` can be found in the Nixpkgs repository. As explained
+
in the introduction of this Python section, a derivation of `toolz` is available
+
for each interpreter version, e.g. `python35.pkgs.toolz` refers to the `toolz`
+
derivation corresponding to the CPython 3.5 interpreter.
The above example works when you're directly working on
`pkgs/top-level/python-packages.nix` in the Nixpkgs repository. Often though,
+
you will want to test a Nix expression outside of the Nixpkgs tree.
+
The following expression creates a derivation for the `toolz` package,
+
and adds it along with a `numpy` package to a Python environment.
with import <nixpkgs> {};
+
my_toolz = python35.pkgs.buildPythonPackage rec {
+
name = "${pname}-${version}";
+
src = python35.pkgs.fetchPypi {
+
sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed05afd";
···
+
in python35.withPackages (ps: [ps.numpy my_toolz])
+
Executing `nix-shell` will result in an environment in which you can use
+
Python 3.5 and the `toolz` package. As you can see we had to explicitly mention
+
for which Python version we want to build a package.
+
So, what did we do here? Well, we took the Nix expression that we used earlier
+
to build a Python environment, and said that we wanted to include our own
+
version of `toolz`, named `my_toolz`. To introduce our own package in the scope
+
of `withPackages` we used a `let` expression. You can see that we used
+
`ps.numpy` to select numpy from the nixpkgs package set (`ps`). We did not take
+
`toolz` from the Nixpkgs package set this time, but instead took our own version
+
that we introduced with the `let` expression.
+
#### Handling dependencies
+
Our example, `toolz`, does not have any dependencies on other Python
packages or system libraries. According to the manual, `buildPythonPackage`
uses the arguments `buildInputs` and `propagatedBuildInputs` to specify dependencies. If something is
exclusively a build-time dependency, then the dependency should be included as a
···
### How to solve circular dependencies?
pyrox.dev