pyrox.dev
1# Writing Tests {#sec-writing-nixos-tests}
2
3A NixOS test is a module that has the following structure:
4
5```nix
6{
7
8 # One or more machines:
9 nodes =
10 { machine =
11 { config, pkgs, ... }: { … };
12 machine2 =
13 { config, pkgs, ... }: { … };
14 …
15 };
16
17 testScript =
18 ''
19 Python code…
20 '';
21}
22```
23
24We refer to the whole test above as a test module, whereas the values
25in [`nodes.<name>`](#test-opt-nodes) are NixOS modules themselves.
26
27The option [`testScript`](#test-opt-testScript) is a piece of Python code that executes the
28test (described below). During the test, it will start one or more
29virtual machines, the configuration of which is described by
30the option [`nodes`](#test-opt-nodes).
31
32An example of a single-node test is
33[`login.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/login.nix).
34It only needs a single machine to test whether users can log in
35on the virtual console, whether device ownership is correctly maintained
36when switching between consoles, and so on. An interesting multi-node test is
37[`nfs/simple.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/nfs/simple.nix).
38It uses two client nodes to test correct locking across server crashes.
39
40## Calling a test {#sec-calling-nixos-tests}
41
42Tests are invoked differently depending on whether the test is part of NixOS or lives in a different project.
43
44### Testing within NixOS {#sec-call-nixos-test-in-nixos}
45
46Tests that are part of NixOS are added to [`nixos/tests/all-tests.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/all-tests.nix).
47
48```nix
49 hostname = runTest ./hostname.nix;
50```
51
52Overrides can be added by defining an anonymous module in `all-tests.nix`.
53
54```nix
55 hostname = runTest {
56 imports = [ ./hostname.nix ];
57 defaults.networking.firewall.enable = false;
58 };
59```
60
61You can run a test with attribute name `hostname` in `nixos/tests/all-tests.nix` by invoking:
62
63```shell
64cd /my/git/clone/of/nixpkgs
65nix-build -A nixosTests.hostname
66```
67
68### Testing outside the NixOS project {#sec-call-nixos-test-outside-nixos}
69
70Outside the `nixpkgs` repository, you can instantiate the test by first importing the NixOS library,
71
72```nix
73let nixos-lib = import (nixpkgs + "/nixos/lib") { };
74in
75
76nixos-lib.runTest {
77 imports = [ ./test.nix ];
78 hostPkgs = pkgs; # the Nixpkgs package set used outside the VMs
79 defaults.services.foo.package = mypkg;
80}
81```
82
83`runTest` returns a derivation that runs the test.
84
85## Configuring the nodes {#sec-nixos-test-nodes}
86
87There are a few special NixOS options for test VMs:
88
89`virtualisation.memorySize`
90
91: The memory of the VM in megabytes.
92
93`virtualisation.vlans`
94
95: The virtual networks to which the VM is connected. See
96 [`nat.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/nat.nix)
97 for an example.
98
99`virtualisation.writableStore`
100
101: By default, the Nix store in the VM is not writable. If you enable
102 this option, a writable union file system is mounted on top of the
103 Nix store to make it appear writable. This is necessary for tests
104 that run Nix operations that modify the store.
105
106For more options, see the module
107[`qemu-vm.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/qemu-vm.nix).
108
109The test script is a sequence of Python statements that perform various
110actions, such as starting VMs, executing commands in the VMs, and so on.
111Each virtual machine is represented as an object stored in the variable
112`name` if this is also the identifier of the machine in the declarative
113config. If you specified a node `nodes.machine`, the following example starts the
114machine, waits until it has finished booting, then executes a command
115and checks that the output is more-or-less correct:
116
117```py
118machine.start()
119machine.wait_for_unit("default.target")
120if not "Linux" in machine.succeed("uname"):
121 raise Exception("Wrong OS")
122```
123
124The first line is technically unnecessary; machines are implicitly started
125when you first execute an action on them (such as `wait_for_unit` or
126`succeed`). If you have multiple machines, you can speed up the test by
127starting them in parallel:
128
129```py
130start_all()
131```
132
133If the hostname of a node contains characters that can't be used in a
134Python variable name, those characters will be replaced with
135underscores in the variable name, so `nodes.machine-a` will be exposed
136to Python as `machine_a`.
137
138## Machine objects {#ssec-machine-objects}
139
140The following methods are available on machine objects:
141
142@PYTHON_MACHINE_METHODS@
143
144To test user units declared by `systemd.user.services` the optional
145`user` argument can be used:
146
147```py
148machine.start()
149machine.wait_for_x()
150machine.wait_for_unit("xautolock.service", "x-session-user")
151```
152
153This applies to `systemctl`, `get_unit_info`, `wait_for_unit`,
154`start_job` and `stop_job`.
155
156For faster dev cycles it's also possible to disable the code-linters
157(this shouldn't be committed though):
158
159```nix
160{
161 skipLint = true;
162 nodes.machine =
163 { config, pkgs, ... }:
164 { configuration…
165 };
166
167 testScript =
168 ''
169 Python code…
170 '';
171}
172```
173
174This will produce a Nix warning at evaluation time. To fully disable the
175linter, wrap the test script in comment directives to disable the Black
176linter directly (again, don't commit this within the Nixpkgs
177repository):
178
179```nix
180 testScript =
181 ''
182 # fmt: off
183 Python code…
184 # fmt: on
185 '';
186```
187
188Similarly, the type checking of test scripts can be disabled in the following
189way:
190
191```nix
192{
193 skipTypeCheck = true;
194 nodes.machine =
195 { config, pkgs, ... }:
196 { configuration…
197 };
198}
199```
200
201## Failing tests early {#ssec-failing-tests-early}
202
203To fail tests early when certain invariants are no longer met (instead of waiting for the build to time out), the decorator `polling_condition` is provided. For example, if we are testing a program `foo` that should not quit after being started, we might write the following:
204
205```py
206@polling_condition
207def foo_running():
208 machine.succeed("pgrep -x foo")
209
210
211machine.succeed("foo --start")
212machine.wait_until_succeeds("pgrep -x foo")
213
214with foo_running:
215 ... # Put `foo` through its paces
216```
217
218`polling_condition` takes the following (optional) arguments:
219
220`seconds_interval`
221
222: specifies how often the condition should be polled:
223
224```py
225@polling_condition(seconds_interval=10)
226def foo_running():
227 machine.succeed("pgrep -x foo")
228```
229
230`description`
231
232: is used in the log when the condition is checked. If this is not provided, the description is pulled from the docstring of the function. These two are therefore equivalent:
233
234```py
235@polling_condition
236def foo_running():
237 "check that foo is running"
238 machine.succeed("pgrep -x foo")
239```
240
241```py
242@polling_condition(description="check that foo is running")
243def foo_running():
244 machine.succeed("pgrep -x foo")
245```
246
247## Adding Python packages to the test script {#ssec-python-packages-in-test-script}
248
249When additional Python libraries are required in the test script, they can be
250added using the parameter `extraPythonPackages`. For example, you could add
251`numpy` like this:
252
253```nix
254{
255 extraPythonPackages = p: [ p.numpy ];
256
257 nodes = { };
258
259 # Type checking on extra packages doesn't work yet
260 skipTypeCheck = true;
261
262 testScript = ''
263 import numpy as np
264 assert str(np.zeros(4) == "array([0., 0., 0., 0.])")
265 '';
266}
267```
268
269In that case, `numpy` is chosen from the generic `python3Packages`.
270
271## Test Options Reference {#sec-test-options-reference}
272
273The following options can be used when writing tests.
274
275```{=include=} options
276id-prefix: test-opt-
277list-id: test-options-list
278source: @NIXOS_TEST_OPTIONS_JSON@
279```