pyrox.dev
1# Writing Tests {#sec-writing-nixos-tests}
2
3A NixOS test is a module that has the following structure:
4
5```nix
6{
7
8 # One or more machines:
9 nodes = {
10 machine =
11 { config, pkgs, ... }:
12 {
13 # ...
14 };
15 machine2 =
16 { config, pkgs, ... }:
17 {
18 # ...
19 };
20 # …
21 };
22
23 testScript = ''
24 Python code…
25 '';
26}
27```
28
29We refer to the whole test above as a test module, whereas the values
30in [`nodes.<name>`](#test-opt-nodes) are NixOS modules themselves.
31
32The option [`testScript`](#test-opt-testScript) is a piece of Python code that executes the
33test (described below). During the test, it will start one or more
34virtual machines, the configuration of which is described by
35the option [`nodes`](#test-opt-nodes).
36
37An example of a single-node test is
38[`login.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/login.nix).
39It only needs a single machine to test whether users can log in
40on the virtual console, whether device ownership is correctly maintained
41when switching between consoles, and so on. An interesting multi-node test is
42[`nfs/simple.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/nfs/simple.nix).
43It uses two client nodes to test correct locking across server crashes.
44
45## Calling a test {#sec-calling-nixos-tests}
46
47Tests are invoked differently depending on whether the test is part of NixOS or lives in a different project.
48
49### Testing within NixOS {#sec-call-nixos-test-in-nixos}
50
51Tests that are part of NixOS are added to [`nixos/tests/all-tests.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/all-tests.nix).
52
53```nix
54{ hostname = runTest ./hostname.nix; }
55```
56
57Overrides can be added by defining an anonymous module in `all-tests.nix`.
58
59```nix
60{
61 hostname = runTest {
62 imports = [ ./hostname.nix ];
63 defaults.networking.firewall.enable = false;
64 };
65}
66```
67
68You can run a test with attribute name `hostname` in `nixos/tests/all-tests.nix` by invoking:
69
70```shell
71cd /my/git/clone/of/nixpkgs
72nix-build -A nixosTests.hostname
73```
74
75### Testing outside the NixOS project {#sec-call-nixos-test-outside-nixos}
76
77Outside the `nixpkgs` repository, you can use the `runNixOSTest` function from
78`pkgs.testers`:
79
80```nix
81let
82 pkgs = import <nixpkgs> { };
83
84in
85pkgs.testers.runNixOSTest {
86 imports = [ ./test.nix ];
87 defaults.services.foo.package = mypkg;
88}
89```
90
91`runNixOSTest` returns a derivation that runs the test.
92
93## Configuring the nodes {#sec-nixos-test-nodes}
94
95There are a few special NixOS options for test VMs:
96
97`virtualisation.memorySize`
98
99: The memory of the VM in MiB (1024×1024 bytes).
100
101`virtualisation.vlans`
102
103: The virtual networks to which the VM is connected. See
104 [`nat.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/nat.nix)
105 for an example.
106
107`virtualisation.writableStore`
108
109: By default, the Nix store in the VM is not writable. If you enable
110 this option, a writable union file system is mounted on top of the
111 Nix store to make it appear writable. This is necessary for tests
112 that run Nix operations that modify the store.
113
114For more options, see the module
115[`qemu-vm.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/qemu-vm.nix).
116
117The test script is a sequence of Python statements that perform various
118actions, such as starting VMs, executing commands in the VMs, and so on.
119Each virtual machine is represented as an object stored in the variable
120`name` if this is also the identifier of the machine in the declarative
121config. If you specified a node `nodes.machine`, the following example starts the
122machine, waits until it has finished booting, then executes a command
123and checks that the output is more-or-less correct:
124
125```py
126machine.start()
127machine.wait_for_unit("default.target")
128t.assertIn("Linux", machine.succeed("uname"), "Wrong OS")
129```
130
131The first line is technically unnecessary; machines are implicitly started
132when you first execute an action on them (such as `wait_for_unit` or
133`succeed`). If you have multiple machines, you can speed up the test by
134starting them in parallel:
135
136```py
137start_all()
138```
139
140Under the variable `t`, all assertions from [`unittest.TestCase`](https://docs.python.org/3/library/unittest.html) are available.
141
142If the hostname of a node contains characters that can't be used in a
143Python variable name, those characters will be replaced with
144underscores in the variable name, so `nodes.machine-a` will be exposed
145to Python as `machine_a`.
146
147## Machine objects {#ssec-machine-objects}
148
149The following methods are available on machine objects:
150
151@PYTHON_MACHINE_METHODS@
152
153To test user units declared by `systemd.user.services` the optional
154`user` argument can be used:
155
156```py
157machine.start()
158machine.wait_for_x()
159machine.wait_for_unit("xautolock.service", "x-session-user")
160```
161
162This applies to `systemctl`, `get_unit_info`, `wait_for_unit`,
163`start_job` and `stop_job`.
164
165For faster dev cycles it's also possible to disable the code-linters
166(this shouldn't be committed though):
167
168```nix
169{
170 skipLint = true;
171 nodes.machine =
172 { config, pkgs, ... }:
173 {
174 # configuration…
175 };
176
177 testScript = ''
178 Python code…
179 '';
180}
181```
182
183This will produce a Nix warning at evaluation time. To fully disable the
184linter, wrap the test script in comment directives to disable the Black
185linter directly (again, don't commit this within the Nixpkgs
186repository):
187
188```nix
189{
190 testScript = ''
191 # fmt: off
192 Python code…
193 # fmt: on
194 '';
195}
196```
197
198Similarly, the type checking of test scripts can be disabled in the following
199way:
200
201```nix
202{
203 skipTypeCheck = true;
204 nodes.machine =
205 { config, pkgs, ... }:
206 {
207 # configuration…
208 };
209}
210```
211
212## Failing tests early {#ssec-failing-tests-early}
213
214To fail tests early when certain invariants are no longer met (instead of waiting for the build to time out), the decorator `polling_condition` is provided. For example, if we are testing a program `foo` that should not quit after being started, we might write the following:
215
216```py
217@polling_condition
218def foo_running():
219 machine.succeed("pgrep -x foo")
220
221
222machine.succeed("foo --start")
223machine.wait_until_succeeds("pgrep -x foo")
224
225with foo_running:
226 ... # Put `foo` through its paces
227```
228
229`polling_condition` takes the following (optional) arguments:
230
231`seconds_interval`
232
233: specifies how often the condition should be polled:
234
235```py
236@polling_condition(seconds_interval=10)
237def foo_running():
238 machine.succeed("pgrep -x foo")
239```
240
241`description`
242
243: is used in the log when the condition is checked. If this is not provided, the description is pulled from the docstring of the function. These two are therefore equivalent:
244
245```py
246@polling_condition
247def foo_running():
248 "check that foo is running"
249 machine.succeed("pgrep -x foo")
250```
251
252```py
253@polling_condition(description="check that foo is running")
254def foo_running():
255 machine.succeed("pgrep -x foo")
256```
257
258## Adding Python packages to the test script {#ssec-python-packages-in-test-script}
259
260When additional Python libraries are required in the test script, they can be
261added using the parameter `extraPythonPackages`. For example, you could add
262`numpy` like this:
263
264```nix
265{
266 extraPythonPackages = p: [ p.numpy ];
267
268 nodes = { };
269
270 # Type checking on extra packages doesn't work yet
271 skipTypeCheck = true;
272
273 testScript = ''
274 import numpy as np
275 assert str(np.zeros(4)) == "[0. 0. 0. 0.]"
276 '';
277}
278```
279
280In that case, `numpy` is chosen from the generic `python3Packages`.
281
282## Overriding a test {#sec-override-nixos-test}
283
284The NixOS test framework returns tests with multiple overriding methods.
285
286`overrideTestDerivation` *function*
287: Like applying `overrideAttrs` on the [test](#test-opt-test) derivation.
288
289 This is a convenience for `extend` with an override on the [`rawTestDerivationArg`](#test-opt-rawTestDerivationArg) option.
290
291 *function*
292 : An extension function, e.g. `finalAttrs: prevAttrs: { /* … */ }`, the result of which is passed to [`mkDerivation`](https://nixos.org/manual/nixpkgs/stable/#sec-using-stdenv).
293 Just as with `overrideAttrs`, an abbreviated form can be used, e.g. `prevAttrs: { /* … */ }` or even `{ /* … */ }`.
294 See [`lib.extends`](https://nixos.org/manual/nixpkgs/stable/#function-library-lib.fixedPoints.extends).
295
296`extendNixOS { module = ` *module* `; specialArgs = ` *specialArgs* `; }`
297: Evaluates the test with additional NixOS modules and/or arguments.
298
299 `module`
300 : A NixOS module to add to all the nodes in the test. Sets test option [`extraBaseModules`](#test-opt-extraBaseModules).
301
302 `specialArgs`
303 : An attribute set of arguments to pass to all NixOS modules. These override the existing arguments, as well as any `_module.args.<name>` that the modules may define. Sets test option [`node.specialArgs`](#test-opt-node.specialArgs).
304
305 This is a convenience function for `extend` that overrides the aforementioned test options.
306
307 :::{.example #ex-nixos-test-extendNixOS}
308
309 # Using extendNixOS in `passthru.tests` to make `(openssh.tests.overrideAttrs f).tests.nixos` coherent
310
311 ```nix
312 mkDerivation (finalAttrs: {
313 # …
314 passthru = {
315 tests = {
316 nixos = nixosTests.openssh.extendNixOS {
317 module = {
318 services.openssh.package = finalAttrs.finalPackage;
319 };
320 };
321 };
322 };
323 })
324 ```
325 :::
326
327`extend { modules = ` *modules* `; specialArgs = ` *specialArgs* `; }`
328: Adds new `nixosTest` modules and/or module arguments to the test, which are evaluated together with the existing modules and [built-in options](#sec-test-options-reference).
329
330 If you're only looking to extend the _NixOS_ configurations of the test, and not something else about the test, you may use the `extendNixOS` convenience function instead.
331
332 `modules`
333 : A list of modules to add to the test. These are added to the existing modules and then [evaluated](https://nixos.org/manual/nixpkgs/stable/index.html#module-system-lib-evalModules) together.
334
335 `specialArgs`
336 : An attribute of arguments to pass to the test. These override the existing arguments, as well as any `_module.args.<name>` that the modules may define. See [`evalModules`/`specialArgs`](https://nixos.org/manual/nixpkgs/stable/#module-system-lib-evalModules-param-specialArgs).
337
338## Test Options Reference {#sec-test-options-reference}
339
340The following options can be used when writing tests.
341
342```{=include=} options
343id-prefix: test-opt-
344list-id: test-options-list
345source: @NIXOS_TEST_OPTIONS_JSON@
346```
347
348## Accessing VMs in the sandbox with SSH {#sec-test-sandbox-breakpoint}
349
350::: {.note}
351For debugging with SSH access into the machines, it's recommended to try using
352[the interactive driver](#sec-running-nixos-tests-interactively) with its
353[SSH backdoor](#sec-nixos-test-ssh-access) first.
354
355This feature is mostly intended to debug flaky test failures that aren't
356reproducible elsewhere.
357:::
358
359As explained in [](#sec-nixos-test-ssh-access), it's possible to configure an
360SSH backdoor based on AF_VSOCK. This can be used to SSH into a VM of a running
361build in a sandbox.
362
363This can be done when something in the test fails, e.g.
364
365```nix
366{
367 nodes.machine = { };
368
369 sshBackdoor.enable = true;
370 enableDebugHook = true;
371
372 testScript = ''
373 start_all()
374 machine.succeed("false") # this will fail
375 '';
376}
377```
378
379For the AF_VSOCK feature to work, `/dev/vhost-vsock` is needed in the sandbox
380which can be done with e.g.
381
382```
383nix-build -A nixosTests.foo --option sandbox-paths /dev/vhost-vsock
384```
385
386This will halt the test execution on a test-failure and print instructions
387on how to enter the sandbox shell of the VM test. Inside, one can log into
388e.g. `machine` with
389
390```
391ssh -F ./ssh_config vsock/3
392```
393
394As described in [](#sec-nixos-test-ssh-access), the numbers for vsock start at
395`3` instead of `1`. So the first VM in the network (sorted alphabetically) can
396be accessed with `vsock/3`.
397
398Alternatively, it's possible to explicitly set a breakpoint with
399`debug.breakpoint()`. This also has the benefit, that one can step through
400`testScript` with `pdb` like this:
401
402```
403$ sudo /nix/store/eeeee-attach <id>
404bash# telnet 127.0.0.1 4444
405pdb$ …
406```