pyrox.dev
1# Writing Tests {#sec-writing-nixos-tests}
2
3A NixOS test is a module that has the following structure:
4
5```nix
6{
7
8 # One or more machines:
9 nodes =
10 { machine =
11 { config, pkgs, ... }: { … };
12 machine2 =
13 { config, pkgs, ... }: { … };
14 …
15 };
16
17 testScript =
18 ''
19 Python code…
20 '';
21}
22```
23
24We refer to the whole test above as a test module, whereas the values
25in [`nodes.<name>`](#test-opt-nodes) are NixOS modules themselves.
26
27The option [`testScript`](#test-opt-testScript) is a piece of Python code that executes the
28test (described below). During the test, it will start one or more
29virtual machines, the configuration of which is described by
30the option [`nodes`](#test-opt-nodes).
31
32An example of a single-node test is
33[`login.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/login.nix).
34It only needs a single machine to test whether users can log in
35on the virtual console, whether device ownership is correctly maintained
36when switching between consoles, and so on. An interesting multi-node test is
37[`nfs/simple.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/nfs/simple.nix).
38It uses two client nodes to test correct locking across server crashes.
39
40## Calling a test {#sec-calling-nixos-tests}
41
42Tests are invoked differently depending on whether the test is part of NixOS or lives in a different project.
43
44### Testing within NixOS {#sec-call-nixos-test-in-nixos}
45
46Tests that are part of NixOS are added to [`nixos/tests/all-tests.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/all-tests.nix).
47
48```nix
49 hostname = runTest ./hostname.nix;
50```
51
52Overrides can be added by defining an anonymous module in `all-tests.nix`.
53
54```nix
55 hostname = runTest {
56 imports = [ ./hostname.nix ];
57 defaults.networking.firewall.enable = false;
58 };
59```
60
61You can run a test with attribute name `hostname` in `nixos/tests/all-tests.nix` by invoking:
62
63```shell
64cd /my/git/clone/of/nixpkgs
65nix-build -A nixosTests.hostname
66```
67
68### Testing outside the NixOS project {#sec-call-nixos-test-outside-nixos}
69
70Outside the `nixpkgs` repository, you can instantiate the test by first importing the NixOS library,
71
72```nix
73let nixos-lib = import (nixpkgs + "/nixos/lib") { };
74in
75
76nixos-lib.runTest {
77 imports = [ ./test.nix ];
78 hostPkgs = pkgs; # the Nixpkgs package set used outside the VMs
79 defaults.services.foo.package = mypkg;
80}
81```
82
83`runTest` returns a derivation that runs the test.
84
85## Configuring the nodes {#sec-nixos-test-nodes}
86
87There are a few special NixOS options for test VMs:
88
89`virtualisation.memorySize`
90
91: The memory of the VM in megabytes.
92
93`virtualisation.vlans`
94
95: The virtual networks to which the VM is connected. See
96 [`nat.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/nat.nix)
97 for an example.
98
99`virtualisation.writableStore`
100
101: By default, the Nix store in the VM is not writable. If you enable
102 this option, a writable union file system is mounted on top of the
103 Nix store to make it appear writable. This is necessary for tests
104 that run Nix operations that modify the store.
105
106For more options, see the module
107[`qemu-vm.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/qemu-vm.nix).
108
109The test script is a sequence of Python statements that perform various
110actions, such as starting VMs, executing commands in the VMs, and so on.
111Each virtual machine is represented as an object stored in the variable
112`name` if this is also the identifier of the machine in the declarative
113config. If you specified a node `nodes.machine`, the following example starts the
114machine, waits until it has finished booting, then executes a command
115and checks that the output is more-or-less correct:
116
117```py
118machine.start()
119machine.wait_for_unit("default.target")
120if not "Linux" in machine.succeed("uname"):
121 raise Exception("Wrong OS")
122```
123
124The first line is technically unnecessary; machines are implicitly started
125when you first execute an action on them (such as `wait_for_unit` or
126`succeed`). If you have multiple machines, you can speed up the test by
127starting them in parallel:
128
129```py
130start_all()
131```
132
133## Machine objects {#ssec-machine-objects}
134
135The following methods are available on machine objects:
136
137`start`
138
139: Start the virtual machine. This method is asynchronous --- it does
140 not wait for the machine to finish booting.
141
142`shutdown`
143
144: Shut down the machine, waiting for the VM to exit.
145
146`crash`
147
148: Simulate a sudden power failure, by telling the VM to exit
149 immediately.
150
151`block`
152
153: Simulate unplugging the Ethernet cable that connects the machine to
154 the other machines.
155
156`unblock`
157
158: Undo the effect of `block`.
159
160`screenshot`
161
162: Take a picture of the display of the virtual machine, in PNG format.
163 The screenshot is linked from the HTML log.
164
165`get_screen_text_variants`
166
167: Return a list of different interpretations of what is currently
168 visible on the machine\'s screen using optical character
169 recognition. The number and order of the interpretations is not
170 specified and is subject to change, but if no exception is raised at
171 least one will be returned.
172
173 ::: {.note}
174 This requires [`enableOCR`](#test-opt-enableOCR) to be set to `true`.
175 :::
176
177`get_screen_text`
178
179: Return a textual representation of what is currently visible on the
180 machine\'s screen using optical character recognition.
181
182 ::: {.note}
183 This requires [`enableOCR`](#test-opt-enableOCR) to be set to `true`.
184 :::
185
186`send_monitor_command`
187
188: Send a command to the QEMU monitor. This is rarely used, but allows
189 doing stuff such as attaching virtual USB disks to a running
190 machine.
191
192`send_key`
193
194: Simulate pressing keys on the virtual keyboard, e.g.,
195 `send_key("ctrl-alt-delete")`.
196
197`send_chars`
198
199: Simulate typing a sequence of characters on the virtual keyboard,
200 e.g., `send_chars("foobar\n")` will type the string `foobar`
201 followed by the Enter key.
202
203`send_console`
204
205: Send keys to the kernel console. This allows interaction with the systemd
206 emergency mode, for example. Takes a string that is sent, e.g.,
207 `send_console("\n\nsystemctl default\n")`.
208
209`execute`
210
211: Execute a shell command, returning a list `(status, stdout)`.
212
213 Commands are run with `set -euo pipefail` set:
214
215 - If several commands are separated by `;` and one fails, the
216 command as a whole will fail.
217
218 - For pipelines, the last non-zero exit status will be returned
219 (if there is one; otherwise zero will be returned).
220
221 - Dereferencing unset variables fails the command.
222
223 - It will wait for stdout to be closed.
224
225 If the command detaches, it must close stdout, as `execute` will wait
226 for this to consume all output reliably. This can be achieved by
227 redirecting stdout to stderr `>&2`, to `/dev/console`, `/dev/null` or
228 a file. Examples of detaching commands are `sleep 365d &`, where the
229 shell forks a new process that can write to stdout and `xclip -i`, where
230 the `xclip` command itself forks without closing stdout.
231
232 Takes an optional parameter `check_return` that defaults to `True`.
233 Setting this parameter to `False` will not check for the return code
234 and return -1 instead. This can be used for commands that shut down
235 the VM and would therefore break the pipe that would be used for
236 retrieving the return code.
237
238 A timeout for the command can be specified (in seconds) using the optional
239 `timeout` parameter, e.g., `execute(cmd, timeout=10)` or
240 `execute(cmd, timeout=None)`. The default is 900 seconds.
241
242`succeed`
243
244: Execute a shell command, raising an exception if the exit status is
245 not zero, otherwise returning the standard output. Similar to `execute`,
246 except that the timeout is `None` by default. See `execute` for details on
247 command execution.
248
249`fail`
250
251: Like `succeed`, but raising an exception if the command returns a zero
252 status.
253
254`wait_until_succeeds`
255
256: Repeat a shell command with 1-second intervals until it succeeds.
257 Has a default timeout of 900 seconds which can be modified, e.g.
258 `wait_until_succeeds(cmd, timeout=10)`. See `execute` for details on
259 command execution.
260
261`wait_until_fails`
262
263: Like `wait_until_succeeds`, but repeating the command until it fails.
264
265`wait_for_unit`
266
267: Wait until the specified systemd unit has reached the "active"
268 state.
269
270`wait_for_file`
271
272: Wait until the specified file exists.
273
274`wait_for_open_port`
275
276: Wait until a process is listening on the given TCP port (on
277 `localhost`, at least).
278
279`wait_for_closed_port`
280
281: Wait until nobody is listening on the given TCP port.
282
283`wait_for_x`
284
285: Wait until the X11 server is accepting connections.
286
287`wait_for_text`
288
289: Wait until the supplied regular expressions matches the textual
290 contents of the screen by using optical character recognition (see
291 `get_screen_text` and `get_screen_text_variants`).
292
293 ::: {.note}
294 This requires [`enableOCR`](#test-opt-enableOCR) to be set to `true`.
295 :::
296
297`wait_for_console_text`
298
299: Wait until the supplied regular expressions match a line of the
300 serial console output. This method is useful when OCR is not
301 possibile or accurate enough.
302
303`wait_for_window`
304
305: Wait until an X11 window has appeared whose name matches the given
306 regular expression, e.g., `wait_for_window("Terminal")`.
307
308`copy_from_host`
309
310: Copies a file from host to machine, e.g.,
311 `copy_from_host("myfile", "/etc/my/important/file")`.
312
313 The first argument is the file on the host. The file needs to be
314 accessible while building the nix derivation. The second argument is
315 the location of the file on the machine.
316
317`systemctl`
318
319: Runs `systemctl` commands with optional support for
320 `systemctl --user`
321
322 ```py
323 machine.systemctl("list-jobs --no-pager") # runs `systemctl list-jobs --no-pager`
324 machine.systemctl("list-jobs --no-pager", "any-user") # spawns a shell for `any-user` and runs `systemctl --user list-jobs --no-pager`
325 ```
326
327`shell_interact`
328
329: Allows you to directly interact with the guest shell. This should
330 only be used during test development, not in production tests.
331 Killing the interactive session with `Ctrl-d` or `Ctrl-c` also ends
332 the guest session.
333
334`console_interact`
335
336: Allows you to directly interact with QEMU's stdin. This should
337 only be used during test development, not in production tests.
338 Output from QEMU is only read line-wise. `Ctrl-c` kills QEMU and
339 `Ctrl-d` closes console and returns to the test runner.
340
341To test user units declared by `systemd.user.services` the optional
342`user` argument can be used:
343
344```py
345machine.start()
346machine.wait_for_x()
347machine.wait_for_unit("xautolock.service", "x-session-user")
348```
349
350This applies to `systemctl`, `get_unit_info`, `wait_for_unit`,
351`start_job` and `stop_job`.
352
353For faster dev cycles it\'s also possible to disable the code-linters
354(this shouldn\'t be commited though):
355
356```nix
357{
358 skipLint = true;
359 nodes.machine =
360 { config, pkgs, ... }:
361 { configuration…
362 };
363
364 testScript =
365 ''
366 Python code…
367 '';
368}
369```
370
371This will produce a Nix warning at evaluation time. To fully disable the
372linter, wrap the test script in comment directives to disable the Black
373linter directly (again, don\'t commit this within the Nixpkgs
374repository):
375
376```nix
377 testScript =
378 ''
379 # fmt: off
380 Python code…
381 # fmt: on
382 '';
383```
384
385Similarly, the type checking of test scripts can be disabled in the following
386way:
387
388```nix
389{
390 skipTypeCheck = true;
391 nodes.machine =
392 { config, pkgs, ... }:
393 { configuration…
394 };
395}
396```
397
398## Failing tests early {#ssec-failing-tests-early}
399
400To fail tests early when certain invariants are no longer met (instead of waiting for the build to time out), the decorator `polling_condition` is provided. For example, if we are testing a program `foo` that should not quit after being started, we might write the following:
401
402```py
403@polling_condition
404def foo_running():
405 machine.succeed("pgrep -x foo")
406
407
408machine.succeed("foo --start")
409machine.wait_until_succeeds("pgrep -x foo")
410
411with foo_running:
412 ... # Put `foo` through its paces
413```
414
415`polling_condition` takes the following (optional) arguments:
416
417`seconds_interval`
418
419:
420 specifies how often the condition should be polled:
421
422```py
423@polling_condition(seconds_interval=10)
424def foo_running():
425 machine.succeed("pgrep -x foo")
426```
427
428`description`
429
430:
431 is used in the log when the condition is checked. If this is not provided, the description is pulled from the docstring of the function. These two are therefore equivalent:
432
433```py
434@polling_condition
435def foo_running():
436 "check that foo is running"
437 machine.succeed("pgrep -x foo")
438```
439
440```py
441@polling_condition(description="check that foo is running")
442def foo_running():
443 machine.succeed("pgrep -x foo")
444```
445
446## Adding Python packages to the test script {#ssec-python-packages-in-test-script}
447
448When additional Python libraries are required in the test script, they can be
449added using the parameter `extraPythonPackages`. For example, you could add
450`numpy` like this:
451
452```nix
453{
454 extraPythonPackages = p: [ p.numpy ];
455
456 nodes = { };
457
458 # Type checking on extra packages doesn't work yet
459 skipTypeCheck = true;
460
461 testScript = ''
462 import numpy as np
463 assert str(np.zeros(4) == "array([0., 0., 0., 0.])")
464 '';
465}
466```
467
468In that case, `numpy` is chosen from the generic `python3Packages`.
469
470## Test Options Reference {#sec-test-options-reference}
471
472The following options can be used when writing tests.
473
474```{=docbook}
475<xi:include href="../../generated/test-options-db.xml" xpointer="test-options-list"/>
476```