pyrox.dev
1# Writing Tests {#sec-writing-nixos-tests}
2
3A NixOS test is a module that has the following structure:
4
5```nix
6{
7
8 # One or more machines:
9 nodes =
10 { machine =
11 { config, pkgs, ... }: { /* ... */ };
12 machine2 =
13 { config, pkgs, ... }: { /* ... */ };
14 # …
15 };
16
17 testScript =
18 ''
19 Python code…
20 '';
21}
22```
23
24We refer to the whole test above as a test module, whereas the values
25in [`nodes.<name>`](#test-opt-nodes) are NixOS modules themselves.
26
27The option [`testScript`](#test-opt-testScript) is a piece of Python code that executes the
28test (described below). During the test, it will start one or more
29virtual machines, the configuration of which is described by
30the option [`nodes`](#test-opt-nodes).
31
32An example of a single-node test is
33[`login.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/login.nix).
34It only needs a single machine to test whether users can log in
35on the virtual console, whether device ownership is correctly maintained
36when switching between consoles, and so on. An interesting multi-node test is
37[`nfs/simple.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/nfs/simple.nix).
38It uses two client nodes to test correct locking across server crashes.
39
40## Calling a test {#sec-calling-nixos-tests}
41
42Tests are invoked differently depending on whether the test is part of NixOS or lives in a different project.
43
44### Testing within NixOS {#sec-call-nixos-test-in-nixos}
45
46Tests that are part of NixOS are added to [`nixos/tests/all-tests.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/all-tests.nix).
47
48```nix
49{
50 hostname = runTest ./hostname.nix;
51}
52```
53
54Overrides can be added by defining an anonymous module in `all-tests.nix`.
55
56```nix
57{
58 hostname = runTest {
59 imports = [ ./hostname.nix ];
60 defaults.networking.firewall.enable = false;
61 };
62}
63```
64
65You can run a test with attribute name `hostname` in `nixos/tests/all-tests.nix` by invoking:
66
67```shell
68cd /my/git/clone/of/nixpkgs
69nix-build -A nixosTests.hostname
70```
71
72### Testing outside the NixOS project {#sec-call-nixos-test-outside-nixos}
73
74Outside the `nixpkgs` repository, you can instantiate the test by first importing the NixOS library,
75
76```nix
77let nixos-lib = import (nixpkgs + "/nixos/lib") { };
78in
79
80nixos-lib.runTest {
81 imports = [ ./test.nix ];
82 hostPkgs = pkgs; # the Nixpkgs package set used outside the VMs
83 defaults.services.foo.package = mypkg;
84}
85```
86
87`runTest` returns a derivation that runs the test.
88
89## Configuring the nodes {#sec-nixos-test-nodes}
90
91There are a few special NixOS options for test VMs:
92
93`virtualisation.memorySize`
94
95: The memory of the VM in megabytes.
96
97`virtualisation.vlans`
98
99: The virtual networks to which the VM is connected. See
100 [`nat.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/nat.nix)
101 for an example.
102
103`virtualisation.writableStore`
104
105: By default, the Nix store in the VM is not writable. If you enable
106 this option, a writable union file system is mounted on top of the
107 Nix store to make it appear writable. This is necessary for tests
108 that run Nix operations that modify the store.
109
110For more options, see the module
111[`qemu-vm.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/qemu-vm.nix).
112
113The test script is a sequence of Python statements that perform various
114actions, such as starting VMs, executing commands in the VMs, and so on.
115Each virtual machine is represented as an object stored in the variable
116`name` if this is also the identifier of the machine in the declarative
117config. If you specified a node `nodes.machine`, the following example starts the
118machine, waits until it has finished booting, then executes a command
119and checks that the output is more-or-less correct:
120
121```py
122machine.start()
123machine.wait_for_unit("default.target")
124if not "Linux" in machine.succeed("uname"):
125 raise Exception("Wrong OS")
126```
127
128The first line is technically unnecessary; machines are implicitly started
129when you first execute an action on them (such as `wait_for_unit` or
130`succeed`). If you have multiple machines, you can speed up the test by
131starting them in parallel:
132
133```py
134start_all()
135```
136
137If the hostname of a node contains characters that can't be used in a
138Python variable name, those characters will be replaced with
139underscores in the variable name, so `nodes.machine-a` will be exposed
140to Python as `machine_a`.
141
142## Machine objects {#ssec-machine-objects}
143
144The following methods are available on machine objects:
145
146@PYTHON_MACHINE_METHODS@
147
148To test user units declared by `systemd.user.services` the optional
149`user` argument can be used:
150
151```py
152machine.start()
153machine.wait_for_x()
154machine.wait_for_unit("xautolock.service", "x-session-user")
155```
156
157This applies to `systemctl`, `get_unit_info`, `wait_for_unit`,
158`start_job` and `stop_job`.
159
160For faster dev cycles it's also possible to disable the code-linters
161(this shouldn't be committed though):
162
163```nix
164{
165 skipLint = true;
166 nodes.machine =
167 { config, pkgs, ... }:
168 { # configuration…
169 };
170
171 testScript =
172 ''
173 Python code…
174 '';
175}
176```
177
178This will produce a Nix warning at evaluation time. To fully disable the
179linter, wrap the test script in comment directives to disable the Black
180linter directly (again, don't commit this within the Nixpkgs
181repository):
182
183```nix
184{
185 testScript =
186 ''
187 # fmt: off
188 Python code…
189 # fmt: on
190 '';
191}
192```
193
194Similarly, the type checking of test scripts can be disabled in the following
195way:
196
197```nix
198{
199 skipTypeCheck = true;
200 nodes.machine =
201 { config, pkgs, ... }:
202 { # configuration…
203 };
204}
205```
206
207## Failing tests early {#ssec-failing-tests-early}
208
209To fail tests early when certain invariants are no longer met (instead of waiting for the build to time out), the decorator `polling_condition` is provided. For example, if we are testing a program `foo` that should not quit after being started, we might write the following:
210
211```py
212@polling_condition
213def foo_running():
214 machine.succeed("pgrep -x foo")
215
216
217machine.succeed("foo --start")
218machine.wait_until_succeeds("pgrep -x foo")
219
220with foo_running:
221 ... # Put `foo` through its paces
222```
223
224`polling_condition` takes the following (optional) arguments:
225
226`seconds_interval`
227
228: specifies how often the condition should be polled:
229
230```py
231@polling_condition(seconds_interval=10)
232def foo_running():
233 machine.succeed("pgrep -x foo")
234```
235
236`description`
237
238: is used in the log when the condition is checked. If this is not provided, the description is pulled from the docstring of the function. These two are therefore equivalent:
239
240```py
241@polling_condition
242def foo_running():
243 "check that foo is running"
244 machine.succeed("pgrep -x foo")
245```
246
247```py
248@polling_condition(description="check that foo is running")
249def foo_running():
250 machine.succeed("pgrep -x foo")
251```
252
253## Adding Python packages to the test script {#ssec-python-packages-in-test-script}
254
255When additional Python libraries are required in the test script, they can be
256added using the parameter `extraPythonPackages`. For example, you could add
257`numpy` like this:
258
259```nix
260{
261 extraPythonPackages = p: [ p.numpy ];
262
263 nodes = { };
264
265 # Type checking on extra packages doesn't work yet
266 skipTypeCheck = true;
267
268 testScript = ''
269 import numpy as np
270 assert str(np.zeros(4)) == "[0. 0. 0. 0.]"
271 '';
272}
273```
274
275In that case, `numpy` is chosen from the generic `python3Packages`.
276
277## Test Options Reference {#sec-test-options-reference}
278
279The following options can be used when writing tests.
280
281```{=include=} options
282id-prefix: test-opt-
283list-id: test-options-list
284source: @NIXOS_TEST_OPTIONS_JSON@
285```