pyrox.dev
1# Writing Tests {#sec-writing-nixos-tests}
2
3A NixOS test is a module that has the following structure:
4
5```nix
6{
7
8 # One or more machines:
9 nodes =
10 { machine =
11 { config, pkgs, ... }: { /* ... */ };
12 machine2 =
13 { config, pkgs, ... }: { /* ... */ };
14 # …
15 };
16
17 testScript =
18 ''
19 Python code…
20 '';
21}
22```
23
24We refer to the whole test above as a test module, whereas the values
25in [`nodes.<name>`](#test-opt-nodes) are NixOS modules themselves.
26
27The option [`testScript`](#test-opt-testScript) is a piece of Python code that executes the
28test (described below). During the test, it will start one or more
29virtual machines, the configuration of which is described by
30the option [`nodes`](#test-opt-nodes).
31
32An example of a single-node test is
33[`login.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/login.nix).
34It only needs a single machine to test whether users can log in
35on the virtual console, whether device ownership is correctly maintained
36when switching between consoles, and so on. An interesting multi-node test is
37[`nfs/simple.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/nfs/simple.nix).
38It uses two client nodes to test correct locking across server crashes.
39
40## Calling a test {#sec-calling-nixos-tests}
41
42Tests are invoked differently depending on whether the test is part of NixOS or lives in a different project.
43
44### Testing within NixOS {#sec-call-nixos-test-in-nixos}
45
46Tests that are part of NixOS are added to [`nixos/tests/all-tests.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/all-tests.nix).
47
48```nix
49{
50 hostname = runTest ./hostname.nix;
51}
52```
53
54Overrides can be added by defining an anonymous module in `all-tests.nix`.
55
56```nix
57{
58 hostname = runTest {
59 imports = [ ./hostname.nix ];
60 defaults.networking.firewall.enable = false;
61 };
62}
63```
64
65You can run a test with attribute name `hostname` in `nixos/tests/all-tests.nix` by invoking:
66
67```shell
68cd /my/git/clone/of/nixpkgs
69nix-build -A nixosTests.hostname
70```
71
72### Testing outside the NixOS project {#sec-call-nixos-test-outside-nixos}
73
74Outside the `nixpkgs` repository, you can use the `runNixOSTest` function from
75`pkgs.testers`:
76
77```nix
78let pkgs = import <nixpkgs> {};
79in
80
81pkgs.testers.runNixOSTest {
82 imports = [ ./test.nix ];
83 defaults.services.foo.package = mypkg;
84}
85```
86
87`runNixOSTest` returns a derivation that runs the test.
88
89## Configuring the nodes {#sec-nixos-test-nodes}
90
91There are a few special NixOS options for test VMs:
92
93`virtualisation.memorySize`
94
95: The memory of the VM in megabytes.
96
97`virtualisation.vlans`
98
99: The virtual networks to which the VM is connected. See
100 [`nat.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/nat.nix)
101 for an example.
102
103`virtualisation.writableStore`
104
105: By default, the Nix store in the VM is not writable. If you enable
106 this option, a writable union file system is mounted on top of the
107 Nix store to make it appear writable. This is necessary for tests
108 that run Nix operations that modify the store.
109
110For more options, see the module
111[`qemu-vm.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/qemu-vm.nix).
112
113The test script is a sequence of Python statements that perform various
114actions, such as starting VMs, executing commands in the VMs, and so on.
115Each virtual machine is represented as an object stored in the variable
116`name` if this is also the identifier of the machine in the declarative
117config. If you specified a node `nodes.machine`, the following example starts the
118machine, waits until it has finished booting, then executes a command
119and checks that the output is more-or-less correct:
120
121```py
122machine.start()
123machine.wait_for_unit("default.target")
124t.assertIn("Linux", machine.succeed("uname"), "Wrong OS")
125```
126
127The first line is technically unnecessary; machines are implicitly started
128when you first execute an action on them (such as `wait_for_unit` or
129`succeed`). If you have multiple machines, you can speed up the test by
130starting them in parallel:
131
132```py
133start_all()
134```
135
136Under the variable `t`, all assertions from [`unittest.TestCase`](https://docs.python.org/3/library/unittest.html) are available.
137
138If the hostname of a node contains characters that can't be used in a
139Python variable name, those characters will be replaced with
140underscores in the variable name, so `nodes.machine-a` will be exposed
141to Python as `machine_a`.
142
143## Machine objects {#ssec-machine-objects}
144
145The following methods are available on machine objects:
146
147@PYTHON_MACHINE_METHODS@
148
149To test user units declared by `systemd.user.services` the optional
150`user` argument can be used:
151
152```py
153machine.start()
154machine.wait_for_x()
155machine.wait_for_unit("xautolock.service", "x-session-user")
156```
157
158This applies to `systemctl`, `get_unit_info`, `wait_for_unit`,
159`start_job` and `stop_job`.
160
161For faster dev cycles it's also possible to disable the code-linters
162(this shouldn't be committed though):
163
164```nix
165{
166 skipLint = true;
167 nodes.machine =
168 { config, pkgs, ... }:
169 { # configuration…
170 };
171
172 testScript =
173 ''
174 Python code…
175 '';
176}
177```
178
179This will produce a Nix warning at evaluation time. To fully disable the
180linter, wrap the test script in comment directives to disable the Black
181linter directly (again, don't commit this within the Nixpkgs
182repository):
183
184```nix
185{
186 testScript =
187 ''
188 # fmt: off
189 Python code…
190 # fmt: on
191 '';
192}
193```
194
195Similarly, the type checking of test scripts can be disabled in the following
196way:
197
198```nix
199{
200 skipTypeCheck = true;
201 nodes.machine =
202 { config, pkgs, ... }:
203 { # configuration…
204 };
205}
206```
207
208## Failing tests early {#ssec-failing-tests-early}
209
210To fail tests early when certain invariants are no longer met (instead of waiting for the build to time out), the decorator `polling_condition` is provided. For example, if we are testing a program `foo` that should not quit after being started, we might write the following:
211
212```py
213@polling_condition
214def foo_running():
215 machine.succeed("pgrep -x foo")
216
217
218machine.succeed("foo --start")
219machine.wait_until_succeeds("pgrep -x foo")
220
221with foo_running:
222 ... # Put `foo` through its paces
223```
224
225`polling_condition` takes the following (optional) arguments:
226
227`seconds_interval`
228
229: specifies how often the condition should be polled:
230
231```py
232@polling_condition(seconds_interval=10)
233def foo_running():
234 machine.succeed("pgrep -x foo")
235```
236
237`description`
238
239: is used in the log when the condition is checked. If this is not provided, the description is pulled from the docstring of the function. These two are therefore equivalent:
240
241```py
242@polling_condition
243def foo_running():
244 "check that foo is running"
245 machine.succeed("pgrep -x foo")
246```
247
248```py
249@polling_condition(description="check that foo is running")
250def foo_running():
251 machine.succeed("pgrep -x foo")
252```
253
254## Adding Python packages to the test script {#ssec-python-packages-in-test-script}
255
256When additional Python libraries are required in the test script, they can be
257added using the parameter `extraPythonPackages`. For example, you could add
258`numpy` like this:
259
260```nix
261{
262 extraPythonPackages = p: [ p.numpy ];
263
264 nodes = { };
265
266 # Type checking on extra packages doesn't work yet
267 skipTypeCheck = true;
268
269 testScript = ''
270 import numpy as np
271 assert str(np.zeros(4)) == "[0. 0. 0. 0.]"
272 '';
273}
274```
275
276In that case, `numpy` is chosen from the generic `python3Packages`.
277
278## Test Options Reference {#sec-test-options-reference}
279
280The following options can be used when writing tests.
281
282```{=include=} options
283id-prefix: test-opt-
284list-id: test-options-list
285source: @NIXOS_TEST_OPTIONS_JSON@
286```