pyrox.dev
1# Writing Tests {#sec-writing-nixos-tests}
2
3A NixOS test is a module that has the following structure:
4
5```nix
6{
7
8 # One or more machines:
9 nodes =
10 { machine =
11 { config, pkgs, ... }: { … };
12 machine2 =
13 { config, pkgs, ... }: { … };
14 …
15 };
16
17 testScript =
18 ''
19 Python code…
20 '';
21}
22```
23
24We refer to the whole test above as a test module, whereas the values
25in [`nodes.<name>`](#test-opt-nodes) are NixOS modules themselves.
26
27The option [`testScript`](#test-opt-testScript) is a piece of Python code that executes the
28test (described below). During the test, it will start one or more
29virtual machines, the configuration of which is described by
30the option [`nodes`](#test-opt-nodes).
31
32An example of a single-node test is
33[`login.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/login.nix).
34It only needs a single machine to test whether users can log in
35on the virtual console, whether device ownership is correctly maintained
36when switching between consoles, and so on. An interesting multi-node test is
37[`nfs/simple.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/nfs/simple.nix).
38It uses two client nodes to test correct locking across server crashes.
39
40## Calling a test {#sec-calling-nixos-tests}
41
42Tests are invoked differently depending on whether the test is part of NixOS or lives in a different project.
43
44### Testing within NixOS {#sec-call-nixos-test-in-nixos}
45
46Tests that are part of NixOS are added to [`nixos/tests/all-tests.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/all-tests.nix).
47
48```nix
49 hostname = runTest ./hostname.nix;
50```
51
52Overrides can be added by defining an anonymous module in `all-tests.nix`.
53
54```nix
55 hostname = runTest {
56 imports = [ ./hostname.nix ];
57 defaults.networking.firewall.enable = false;
58 };
59```
60
61You can run a test with attribute name `hostname` in `nixos/tests/all-tests.nix` by invoking:
62
63```shell
64cd /my/git/clone/of/nixpkgs
65nix-build -A nixosTests.hostname
66```
67
68### Testing outside the NixOS project {#sec-call-nixos-test-outside-nixos}
69
70Outside the `nixpkgs` repository, you can instantiate the test by first importing the NixOS library,
71
72```nix
73let nixos-lib = import (nixpkgs + "/nixos/lib") { };
74in
75
76nixos-lib.runTest {
77 imports = [ ./test.nix ];
78 hostPkgs = pkgs; # the Nixpkgs package set used outside the VMs
79 defaults.services.foo.package = mypkg;
80}
81```
82
83`runTest` returns a derivation that runs the test.
84
85## Configuring the nodes {#sec-nixos-test-nodes}
86
87There are a few special NixOS options for test VMs:
88
89`virtualisation.memorySize`
90
91: The memory of the VM in megabytes.
92
93`virtualisation.vlans`
94
95: The virtual networks to which the VM is connected. See
96 [`nat.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/nat.nix)
97 for an example.
98
99`virtualisation.writableStore`
100
101: By default, the Nix store in the VM is not writable. If you enable
102 this option, a writable union file system is mounted on top of the
103 Nix store to make it appear writable. This is necessary for tests
104 that run Nix operations that modify the store.
105
106For more options, see the module
107[`qemu-vm.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/qemu-vm.nix).
108
109The test script is a sequence of Python statements that perform various
110actions, such as starting VMs, executing commands in the VMs, and so on.
111Each virtual machine is represented as an object stored in the variable
112`name` if this is also the identifier of the machine in the declarative
113config. If you specified a node `nodes.machine`, the following example starts the
114machine, waits until it has finished booting, then executes a command
115and checks that the output is more-or-less correct:
116
117```py
118machine.start()
119machine.wait_for_unit("default.target")
120if not "Linux" in machine.succeed("uname"):
121 raise Exception("Wrong OS")
122```
123
124The first line is technically unnecessary; machines are implicitly started
125when you first execute an action on them (such as `wait_for_unit` or
126`succeed`). If you have multiple machines, you can speed up the test by
127starting them in parallel:
128
129```py
130start_all()
131```
132
133If the hostname of a node contains characters that can't be used in a
134Python variable name, those characters will be replaced with
135underscores in the variable name, so `nodes.machine-a` will be exposed
136to Python as `machine_a`.
137
138## Machine objects {#ssec-machine-objects}
139
140The following methods are available on machine objects:
141
142`start`
143
144: Start the virtual machine. This method is asynchronous --- it does
145 not wait for the machine to finish booting.
146
147`shutdown`
148
149: Shut down the machine, waiting for the VM to exit.
150
151`crash`
152
153: Simulate a sudden power failure, by telling the VM to exit
154 immediately.
155
156`block`
157
158: Simulate unplugging the Ethernet cable that connects the machine to
159 the other machines.
160
161`unblock`
162
163: Undo the effect of `block`.
164
165`screenshot`
166
167: Take a picture of the display of the virtual machine, in PNG format.
168 The screenshot is linked from the HTML log.
169
170`get_screen_text_variants`
171
172: Return a list of different interpretations of what is currently
173 visible on the machine's screen using optical character
174 recognition. The number and order of the interpretations is not
175 specified and is subject to change, but if no exception is raised at
176 least one will be returned.
177
178 ::: {.note}
179 This requires [`enableOCR`](#test-opt-enableOCR) to be set to `true`.
180 :::
181
182`get_screen_text`
183
184: Return a textual representation of what is currently visible on the
185 machine's screen using optical character recognition.
186
187 ::: {.note}
188 This requires [`enableOCR`](#test-opt-enableOCR) to be set to `true`.
189 :::
190
191`send_monitor_command`
192
193: Send a command to the QEMU monitor. This is rarely used, but allows
194 doing stuff such as attaching virtual USB disks to a running
195 machine.
196
197`send_key`
198
199: Simulate pressing keys on the virtual keyboard, e.g.,
200 `send_key("ctrl-alt-delete")`.
201
202`send_chars`
203
204: Simulate typing a sequence of characters on the virtual keyboard,
205 e.g., `send_chars("foobar\n")` will type the string `foobar`
206 followed by the Enter key.
207
208`send_console`
209
210: Send keys to the kernel console. This allows interaction with the systemd
211 emergency mode, for example. Takes a string that is sent, e.g.,
212 `send_console("\n\nsystemctl default\n")`.
213
214`execute`
215
216: Execute a shell command, returning a list `(status, stdout)`.
217
218 Commands are run with `set -euo pipefail` set:
219
220 - If several commands are separated by `;` and one fails, the
221 command as a whole will fail.
222
223 - For pipelines, the last non-zero exit status will be returned
224 (if there is one; otherwise zero will be returned).
225
226 - Dereferencing unset variables fails the command.
227
228 - It will wait for stdout to be closed.
229
230 If the command detaches, it must close stdout, as `execute` will wait
231 for this to consume all output reliably. This can be achieved by
232 redirecting stdout to stderr `>&2`, to `/dev/console`, `/dev/null` or
233 a file. Examples of detaching commands are `sleep 365d &`, where the
234 shell forks a new process that can write to stdout and `xclip -i`, where
235 the `xclip` command itself forks without closing stdout.
236
237 Takes an optional parameter `check_return` that defaults to `True`.
238 Setting this parameter to `False` will not check for the return code
239 and return -1 instead. This can be used for commands that shut down
240 the VM and would therefore break the pipe that would be used for
241 retrieving the return code.
242
243 A timeout for the command can be specified (in seconds) using the optional
244 `timeout` parameter, e.g., `execute(cmd, timeout=10)` or
245 `execute(cmd, timeout=None)`. The default is 900 seconds.
246
247`succeed`
248
249: Execute a shell command, raising an exception if the exit status is
250 not zero, otherwise returning the standard output. Similar to `execute`,
251 except that the timeout is `None` by default. See `execute` for details on
252 command execution.
253
254`fail`
255
256: Like `succeed`, but raising an exception if the command returns a zero
257 status.
258
259`wait_until_succeeds`
260
261: Repeat a shell command with 1-second intervals until it succeeds.
262 Has a default timeout of 900 seconds which can be modified, e.g.
263 `wait_until_succeeds(cmd, timeout=10)`. See `execute` for details on
264 command execution.
265
266`wait_until_fails`
267
268: Like `wait_until_succeeds`, but repeating the command until it fails.
269
270`wait_for_unit`
271
272: Wait until the specified systemd unit has reached the "active"
273 state.
274
275`wait_for_file`
276
277: Wait until the specified file exists.
278
279`wait_for_open_port`
280
281: Wait until a process is listening on the given TCP port and IP address
282 (default `localhost`).
283
284`wait_for_closed_port`
285
286: Wait until nobody is listening on the given TCP port and IP address
287 (default `localhost`).
288
289`wait_for_x`
290
291: Wait until the X11 server is accepting connections.
292
293`wait_for_text`
294
295: Wait until the supplied regular expressions matches the textual
296 contents of the screen by using optical character recognition (see
297 `get_screen_text` and `get_screen_text_variants`).
298
299 ::: {.note}
300 This requires [`enableOCR`](#test-opt-enableOCR) to be set to `true`.
301 :::
302
303`wait_for_console_text`
304
305: Wait until the supplied regular expressions match a line of the
306 serial console output. This method is useful when OCR is not
307 possible or accurate enough.
308
309`wait_for_window`
310
311: Wait until an X11 window has appeared whose name matches the given
312 regular expression, e.g., `wait_for_window("Terminal")`.
313
314`copy_from_host`
315
316: Copies a file from host to machine, e.g.,
317 `copy_from_host("myfile", "/etc/my/important/file")`.
318
319 The first argument is the file on the host. The file needs to be
320 accessible while building the nix derivation. The second argument is
321 the location of the file on the machine.
322
323`systemctl`
324
325: Runs `systemctl` commands with optional support for
326 `systemctl --user`
327
328 ```py
329 machine.systemctl("list-jobs --no-pager") # runs `systemctl list-jobs --no-pager`
330 machine.systemctl("list-jobs --no-pager", "any-user") # spawns a shell for `any-user` and runs `systemctl --user list-jobs --no-pager`
331 ```
332
333`shell_interact`
334
335: Allows you to directly interact with the guest shell. This should
336 only be used during test development, not in production tests.
337 Killing the interactive session with `Ctrl-d` or `Ctrl-c` also ends
338 the guest session.
339
340`console_interact`
341
342: Allows you to directly interact with QEMU's stdin. This should
343 only be used during test development, not in production tests.
344 Output from QEMU is only read line-wise. `Ctrl-c` kills QEMU and
345 `Ctrl-d` closes console and returns to the test runner.
346
347To test user units declared by `systemd.user.services` the optional
348`user` argument can be used:
349
350```py
351machine.start()
352machine.wait_for_x()
353machine.wait_for_unit("xautolock.service", "x-session-user")
354```
355
356This applies to `systemctl`, `get_unit_info`, `wait_for_unit`,
357`start_job` and `stop_job`.
358
359For faster dev cycles it's also possible to disable the code-linters
360(this shouldn't be committed though):
361
362```nix
363{
364 skipLint = true;
365 nodes.machine =
366 { config, pkgs, ... }:
367 { configuration…
368 };
369
370 testScript =
371 ''
372 Python code…
373 '';
374}
375```
376
377This will produce a Nix warning at evaluation time. To fully disable the
378linter, wrap the test script in comment directives to disable the Black
379linter directly (again, don't commit this within the Nixpkgs
380repository):
381
382```nix
383 testScript =
384 ''
385 # fmt: off
386 Python code…
387 # fmt: on
388 '';
389```
390
391Similarly, the type checking of test scripts can be disabled in the following
392way:
393
394```nix
395{
396 skipTypeCheck = true;
397 nodes.machine =
398 { config, pkgs, ... }:
399 { configuration…
400 };
401}
402```
403
404## Failing tests early {#ssec-failing-tests-early}
405
406To fail tests early when certain invariants are no longer met (instead of waiting for the build to time out), the decorator `polling_condition` is provided. For example, if we are testing a program `foo` that should not quit after being started, we might write the following:
407
408```py
409@polling_condition
410def foo_running():
411 machine.succeed("pgrep -x foo")
412
413
414machine.succeed("foo --start")
415machine.wait_until_succeeds("pgrep -x foo")
416
417with foo_running:
418 ... # Put `foo` through its paces
419```
420
421`polling_condition` takes the following (optional) arguments:
422
423`seconds_interval`
424
425: specifies how often the condition should be polled:
426
427```py
428@polling_condition(seconds_interval=10)
429def foo_running():
430 machine.succeed("pgrep -x foo")
431```
432
433`description`
434
435: is used in the log when the condition is checked. If this is not provided, the description is pulled from the docstring of the function. These two are therefore equivalent:
436
437```py
438@polling_condition
439def foo_running():
440 "check that foo is running"
441 machine.succeed("pgrep -x foo")
442```
443
444```py
445@polling_condition(description="check that foo is running")
446def foo_running():
447 machine.succeed("pgrep -x foo")
448```
449
450## Adding Python packages to the test script {#ssec-python-packages-in-test-script}
451
452When additional Python libraries are required in the test script, they can be
453added using the parameter `extraPythonPackages`. For example, you could add
454`numpy` like this:
455
456```nix
457{
458 extraPythonPackages = p: [ p.numpy ];
459
460 nodes = { };
461
462 # Type checking on extra packages doesn't work yet
463 skipTypeCheck = true;
464
465 testScript = ''
466 import numpy as np
467 assert str(np.zeros(4) == "array([0., 0., 0., 0.])")
468 '';
469}
470```
471
472In that case, `numpy` is chosen from the generic `python3Packages`.
473
474## Test Options Reference {#sec-test-options-reference}
475
476The following options can be used when writing tests.
477
478```{=include=} options
479id-prefix: test-opt-
480list-id: test-options-list
481source: @NIXOS_TEST_OPTIONS_JSON@
482```