pyrox.dev
1<section xmlns="http://docbook.org/ns/docbook" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xi="http://www.w3.org/2001/XInclude" xml:id="sec-writing-nixos-tests">
2 <title>Writing Tests</title>
3 <para>
4 A NixOS test is a module that has the following structure:
5 </para>
6 <programlisting language="bash">
7{
8
9 # One or more machines:
10 nodes =
11 { machine =
12 { config, pkgs, ... }: { … };
13 machine2 =
14 { config, pkgs, ... }: { … };
15 …
16 };
17
18 testScript =
19 ''
20 Python code…
21 '';
22}
23</programlisting>
24 <para>
25 We refer to the whole test above as a test module, whereas the
26 values in
27 <link linkend="test-opt-nodes"><literal>nodes.<name></literal></link>
28 are NixOS modules themselves.
29 </para>
30 <para>
31 The option
32 <link linkend="test-opt-testScript"><literal>testScript</literal></link>
33 is a piece of Python code that executes the test (described below).
34 During the test, it will start one or more virtual machines, the
35 configuration of which is described by the option
36 <link linkend="test-opt-nodes"><literal>nodes</literal></link>.
37 </para>
38 <para>
39 An example of a single-node test is
40 <link xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/login.nix"><literal>login.nix</literal></link>.
41 It only needs a single machine to test whether users can log in on
42 the virtual console, whether device ownership is correctly
43 maintained when switching between consoles, and so on. An
44 interesting multi-node test is
45 <link xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/nfs/simple.nix"><literal>nfs/simple.nix</literal></link>.
46 It uses two client nodes to test correct locking across server
47 crashes.
48 </para>
49 <section xml:id="sec-calling-nixos-tests">
50 <title>Calling a test</title>
51 <para>
52 Tests are invoked differently depending on whether the test is
53 part of NixOS or lives in a different project.
54 </para>
55 <section xml:id="sec-call-nixos-test-in-nixos">
56 <title>Testing within NixOS</title>
57 <para>
58 Tests that are part of NixOS are added to
59 <link xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/all-tests.nix"><literal>nixos/tests/all-tests.nix</literal></link>.
60 </para>
61 <programlisting language="bash">
62 hostname = runTest ./hostname.nix;
63</programlisting>
64 <para>
65 Overrides can be added by defining an anonymous module in
66 <literal>all-tests.nix</literal>.
67 </para>
68 <programlisting language="bash">
69 hostname = runTest {
70 imports = [ ./hostname.nix ];
71 defaults.networking.firewall.enable = false;
72 };
73</programlisting>
74 <para>
75 You can run a test with attribute name
76 <literal>hostname</literal> in
77 <literal>nixos/tests/all-tests.nix</literal> by invoking:
78 </para>
79 <programlisting>
80cd /my/git/clone/of/nixpkgs
81nix-build -A nixosTests.hostname
82</programlisting>
83 </section>
84 <section xml:id="sec-call-nixos-test-outside-nixos">
85 <title>Testing outside the NixOS project</title>
86 <para>
87 Outside the <literal>nixpkgs</literal> repository, you can
88 instantiate the test by first importing the NixOS library,
89 </para>
90 <programlisting language="bash">
91let nixos-lib = import (nixpkgs + "/nixos/lib") { };
92in
93
94nixos-lib.runTest {
95 imports = [ ./test.nix ];
96 hostPkgs = pkgs; # the Nixpkgs package set used outside the VMs
97 defaults.services.foo.package = mypkg;
98}
99</programlisting>
100 <para>
101 <literal>runTest</literal> returns a derivation that runs the
102 test.
103 </para>
104 </section>
105 </section>
106 <section xml:id="sec-nixos-test-nodes">
107 <title>Configuring the nodes</title>
108 <para>
109 There are a few special NixOS options for test VMs:
110 </para>
111 <variablelist>
112 <varlistentry>
113 <term>
114 <literal>virtualisation.memorySize</literal>
115 </term>
116 <listitem>
117 <para>
118 The memory of the VM in megabytes.
119 </para>
120 </listitem>
121 </varlistentry>
122 <varlistentry>
123 <term>
124 <literal>virtualisation.vlans</literal>
125 </term>
126 <listitem>
127 <para>
128 The virtual networks to which the VM is connected. See
129 <link xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/nat.nix"><literal>nat.nix</literal></link>
130 for an example.
131 </para>
132 </listitem>
133 </varlistentry>
134 <varlistentry>
135 <term>
136 <literal>virtualisation.writableStore</literal>
137 </term>
138 <listitem>
139 <para>
140 By default, the Nix store in the VM is not writable. If you
141 enable this option, a writable union file system is mounted
142 on top of the Nix store to make it appear writable. This is
143 necessary for tests that run Nix operations that modify the
144 store.
145 </para>
146 </listitem>
147 </varlistentry>
148 </variablelist>
149 <para>
150 For more options, see the module
151 <link xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/qemu-vm.nix"><literal>qemu-vm.nix</literal></link>.
152 </para>
153 <para>
154 The test script is a sequence of Python statements that perform
155 various actions, such as starting VMs, executing commands in the
156 VMs, and so on. Each virtual machine is represented as an object
157 stored in the variable <literal>name</literal> if this is also the
158 identifier of the machine in the declarative config. If you
159 specified a node <literal>nodes.machine</literal>, the following
160 example starts the machine, waits until it has finished booting,
161 then executes a command and checks that the output is more-or-less
162 correct:
163 </para>
164 <programlisting language="python">
165machine.start()
166machine.wait_for_unit("default.target")
167if not "Linux" in machine.succeed("uname"):
168 raise Exception("Wrong OS")
169</programlisting>
170 <para>
171 The first line is technically unnecessary; machines are implicitly
172 started when you first execute an action on them (such as
173 <literal>wait_for_unit</literal> or <literal>succeed</literal>).
174 If you have multiple machines, you can speed up the test by
175 starting them in parallel:
176 </para>
177 <programlisting language="python">
178start_all()
179</programlisting>
180 </section>
181 <section xml:id="ssec-machine-objects">
182 <title>Machine objects</title>
183 <para>
184 The following methods are available on machine objects:
185 </para>
186 <variablelist>
187 <varlistentry>
188 <term>
189 <literal>start</literal>
190 </term>
191 <listitem>
192 <para>
193 Start the virtual machine. This method is asynchronous — it
194 does not wait for the machine to finish booting.
195 </para>
196 </listitem>
197 </varlistentry>
198 <varlistentry>
199 <term>
200 <literal>shutdown</literal>
201 </term>
202 <listitem>
203 <para>
204 Shut down the machine, waiting for the VM to exit.
205 </para>
206 </listitem>
207 </varlistentry>
208 <varlistentry>
209 <term>
210 <literal>crash</literal>
211 </term>
212 <listitem>
213 <para>
214 Simulate a sudden power failure, by telling the VM to exit
215 immediately.
216 </para>
217 </listitem>
218 </varlistentry>
219 <varlistentry>
220 <term>
221 <literal>block</literal>
222 </term>
223 <listitem>
224 <para>
225 Simulate unplugging the Ethernet cable that connects the
226 machine to the other machines.
227 </para>
228 </listitem>
229 </varlistentry>
230 <varlistentry>
231 <term>
232 <literal>unblock</literal>
233 </term>
234 <listitem>
235 <para>
236 Undo the effect of <literal>block</literal>.
237 </para>
238 </listitem>
239 </varlistentry>
240 <varlistentry>
241 <term>
242 <literal>screenshot</literal>
243 </term>
244 <listitem>
245 <para>
246 Take a picture of the display of the virtual machine, in PNG
247 format. The screenshot is linked from the HTML log.
248 </para>
249 </listitem>
250 </varlistentry>
251 <varlistentry>
252 <term>
253 <literal>get_screen_text_variants</literal>
254 </term>
255 <listitem>
256 <para>
257 Return a list of different interpretations of what is
258 currently visible on the machine's screen using optical
259 character recognition. The number and order of the
260 interpretations is not specified and is subject to change,
261 but if no exception is raised at least one will be returned.
262 </para>
263 <note>
264 <para>
265 This requires
266 <link linkend="test-opt-enableOCR"><literal>enableOCR</literal></link>
267 to be set to <literal>true</literal>.
268 </para>
269 </note>
270 </listitem>
271 </varlistentry>
272 <varlistentry>
273 <term>
274 <literal>get_screen_text</literal>
275 </term>
276 <listitem>
277 <para>
278 Return a textual representation of what is currently visible
279 on the machine's screen using optical character recognition.
280 </para>
281 <note>
282 <para>
283 This requires
284 <link linkend="test-opt-enableOCR"><literal>enableOCR</literal></link>
285 to be set to <literal>true</literal>.
286 </para>
287 </note>
288 </listitem>
289 </varlistentry>
290 <varlistentry>
291 <term>
292 <literal>send_monitor_command</literal>
293 </term>
294 <listitem>
295 <para>
296 Send a command to the QEMU monitor. This is rarely used, but
297 allows doing stuff such as attaching virtual USB disks to a
298 running machine.
299 </para>
300 </listitem>
301 </varlistentry>
302 <varlistentry>
303 <term>
304 <literal>send_key</literal>
305 </term>
306 <listitem>
307 <para>
308 Simulate pressing keys on the virtual keyboard, e.g.,
309 <literal>send_key("ctrl-alt-delete")</literal>.
310 </para>
311 </listitem>
312 </varlistentry>
313 <varlistentry>
314 <term>
315 <literal>send_chars</literal>
316 </term>
317 <listitem>
318 <para>
319 Simulate typing a sequence of characters on the virtual
320 keyboard, e.g.,
321 <literal>send_chars("foobar\n")</literal> will
322 type the string <literal>foobar</literal> followed by the
323 Enter key.
324 </para>
325 </listitem>
326 </varlistentry>
327 <varlistentry>
328 <term>
329 <literal>send_console</literal>
330 </term>
331 <listitem>
332 <para>
333 Send keys to the kernel console. This allows interaction
334 with the systemd emergency mode, for example. Takes a string
335 that is sent, e.g.,
336 <literal>send_console("\n\nsystemctl default\n")</literal>.
337 </para>
338 </listitem>
339 </varlistentry>
340 <varlistentry>
341 <term>
342 <literal>execute</literal>
343 </term>
344 <listitem>
345 <para>
346 Execute a shell command, returning a list
347 <literal>(status, stdout)</literal>.
348 </para>
349 <para>
350 Commands are run with <literal>set -euo pipefail</literal>
351 set:
352 </para>
353 <itemizedlist>
354 <listitem>
355 <para>
356 If several commands are separated by
357 <literal>;</literal> and one fails, the command as a
358 whole will fail.
359 </para>
360 </listitem>
361 <listitem>
362 <para>
363 For pipelines, the last non-zero exit status will be
364 returned (if there is one; otherwise zero will be
365 returned).
366 </para>
367 </listitem>
368 <listitem>
369 <para>
370 Dereferencing unset variables fails the command.
371 </para>
372 </listitem>
373 <listitem>
374 <para>
375 It will wait for stdout to be closed.
376 </para>
377 </listitem>
378 </itemizedlist>
379 <para>
380 If the command detaches, it must close stdout, as
381 <literal>execute</literal> will wait for this to consume all
382 output reliably. This can be achieved by redirecting stdout
383 to stderr <literal>>&2</literal>, to
384 <literal>/dev/console</literal>,
385 <literal>/dev/null</literal> or a file. Examples of
386 detaching commands are <literal>sleep 365d &</literal>,
387 where the shell forks a new process that can write to stdout
388 and <literal>xclip -i</literal>, where the
389 <literal>xclip</literal> command itself forks without
390 closing stdout.
391 </para>
392 <para>
393 Takes an optional parameter <literal>check_return</literal>
394 that defaults to <literal>True</literal>. Setting this
395 parameter to <literal>False</literal> will not check for the
396 return code and return -1 instead. This can be used for
397 commands that shut down the VM and would therefore break the
398 pipe that would be used for retrieving the return code.
399 </para>
400 <para>
401 A timeout for the command can be specified (in seconds)
402 using the optional <literal>timeout</literal> parameter,
403 e.g., <literal>execute(cmd, timeout=10)</literal> or
404 <literal>execute(cmd, timeout=None)</literal>. The default
405 is 900 seconds.
406 </para>
407 </listitem>
408 </varlistentry>
409 <varlistentry>
410 <term>
411 <literal>succeed</literal>
412 </term>
413 <listitem>
414 <para>
415 Execute a shell command, raising an exception if the exit
416 status is not zero, otherwise returning the standard output.
417 Similar to <literal>execute</literal>, except that the
418 timeout is <literal>None</literal> by default. See
419 <literal>execute</literal> for details on command execution.
420 </para>
421 </listitem>
422 </varlistentry>
423 <varlistentry>
424 <term>
425 <literal>fail</literal>
426 </term>
427 <listitem>
428 <para>
429 Like <literal>succeed</literal>, but raising an exception if
430 the command returns a zero status.
431 </para>
432 </listitem>
433 </varlistentry>
434 <varlistentry>
435 <term>
436 <literal>wait_until_succeeds</literal>
437 </term>
438 <listitem>
439 <para>
440 Repeat a shell command with 1-second intervals until it
441 succeeds. Has a default timeout of 900 seconds which can be
442 modified, e.g.
443 <literal>wait_until_succeeds(cmd, timeout=10)</literal>. See
444 <literal>execute</literal> for details on command execution.
445 </para>
446 </listitem>
447 </varlistentry>
448 <varlistentry>
449 <term>
450 <literal>wait_until_fails</literal>
451 </term>
452 <listitem>
453 <para>
454 Like <literal>wait_until_succeeds</literal>, but repeating
455 the command until it fails.
456 </para>
457 </listitem>
458 </varlistentry>
459 <varlistentry>
460 <term>
461 <literal>wait_for_unit</literal>
462 </term>
463 <listitem>
464 <para>
465 Wait until the specified systemd unit has reached the
466 <quote>active</quote> state.
467 </para>
468 </listitem>
469 </varlistentry>
470 <varlistentry>
471 <term>
472 <literal>wait_for_file</literal>
473 </term>
474 <listitem>
475 <para>
476 Wait until the specified file exists.
477 </para>
478 </listitem>
479 </varlistentry>
480 <varlistentry>
481 <term>
482 <literal>wait_for_open_port</literal>
483 </term>
484 <listitem>
485 <para>
486 Wait until a process is listening on the given TCP port (on
487 <literal>localhost</literal>, at least).
488 </para>
489 </listitem>
490 </varlistentry>
491 <varlistentry>
492 <term>
493 <literal>wait_for_closed_port</literal>
494 </term>
495 <listitem>
496 <para>
497 Wait until nobody is listening on the given TCP port.
498 </para>
499 </listitem>
500 </varlistentry>
501 <varlistentry>
502 <term>
503 <literal>wait_for_x</literal>
504 </term>
505 <listitem>
506 <para>
507 Wait until the X11 server is accepting connections.
508 </para>
509 </listitem>
510 </varlistentry>
511 <varlistentry>
512 <term>
513 <literal>wait_for_text</literal>
514 </term>
515 <listitem>
516 <para>
517 Wait until the supplied regular expressions matches the
518 textual contents of the screen by using optical character
519 recognition (see <literal>get_screen_text</literal> and
520 <literal>get_screen_text_variants</literal>).
521 </para>
522 <note>
523 <para>
524 This requires
525 <link linkend="test-opt-enableOCR"><literal>enableOCR</literal></link>
526 to be set to <literal>true</literal>.
527 </para>
528 </note>
529 </listitem>
530 </varlistentry>
531 <varlistentry>
532 <term>
533 <literal>wait_for_console_text</literal>
534 </term>
535 <listitem>
536 <para>
537 Wait until the supplied regular expressions match a line of
538 the serial console output. This method is useful when OCR is
539 not possibile or accurate enough.
540 </para>
541 </listitem>
542 </varlistentry>
543 <varlistentry>
544 <term>
545 <literal>wait_for_window</literal>
546 </term>
547 <listitem>
548 <para>
549 Wait until an X11 window has appeared whose name matches the
550 given regular expression, e.g.,
551 <literal>wait_for_window("Terminal")</literal>.
552 </para>
553 </listitem>
554 </varlistentry>
555 <varlistentry>
556 <term>
557 <literal>copy_from_host</literal>
558 </term>
559 <listitem>
560 <para>
561 Copies a file from host to machine, e.g.,
562 <literal>copy_from_host("myfile", "/etc/my/important/file")</literal>.
563 </para>
564 <para>
565 The first argument is the file on the host. The file needs
566 to be accessible while building the nix derivation. The
567 second argument is the location of the file on the machine.
568 </para>
569 </listitem>
570 </varlistentry>
571 <varlistentry>
572 <term>
573 <literal>systemctl</literal>
574 </term>
575 <listitem>
576 <para>
577 Runs <literal>systemctl</literal> commands with optional
578 support for <literal>systemctl --user</literal>
579 </para>
580 <programlisting language="python">
581machine.systemctl("list-jobs --no-pager") # runs `systemctl list-jobs --no-pager`
582machine.systemctl("list-jobs --no-pager", "any-user") # spawns a shell for `any-user` and runs `systemctl --user list-jobs --no-pager`
583</programlisting>
584 </listitem>
585 </varlistentry>
586 <varlistentry>
587 <term>
588 <literal>shell_interact</literal>
589 </term>
590 <listitem>
591 <para>
592 Allows you to directly interact with the guest shell. This
593 should only be used during test development, not in
594 production tests. Killing the interactive session with
595 <literal>Ctrl-d</literal> or <literal>Ctrl-c</literal> also
596 ends the guest session.
597 </para>
598 </listitem>
599 </varlistentry>
600 <varlistentry>
601 <term>
602 <literal>console_interact</literal>
603 </term>
604 <listitem>
605 <para>
606 Allows you to directly interact with QEMU’s stdin. This
607 should only be used during test development, not in
608 production tests. Output from QEMU is only read line-wise.
609 <literal>Ctrl-c</literal> kills QEMU and
610 <literal>Ctrl-d</literal> closes console and returns to the
611 test runner.
612 </para>
613 </listitem>
614 </varlistentry>
615 </variablelist>
616 <para>
617 To test user units declared by
618 <literal>systemd.user.services</literal> the optional
619 <literal>user</literal> argument can be used:
620 </para>
621 <programlisting language="python">
622machine.start()
623machine.wait_for_x()
624machine.wait_for_unit("xautolock.service", "x-session-user")
625</programlisting>
626 <para>
627 This applies to <literal>systemctl</literal>,
628 <literal>get_unit_info</literal>,
629 <literal>wait_for_unit</literal>, <literal>start_job</literal> and
630 <literal>stop_job</literal>.
631 </para>
632 <para>
633 For faster dev cycles it's also possible to disable the
634 code-linters (this shouldn't be commited though):
635 </para>
636 <programlisting language="bash">
637{
638 skipLint = true;
639 nodes.machine =
640 { config, pkgs, ... }:
641 { configuration…
642 };
643
644 testScript =
645 ''
646 Python code…
647 '';
648}
649</programlisting>
650 <para>
651 This will produce a Nix warning at evaluation time. To fully
652 disable the linter, wrap the test script in comment directives to
653 disable the Black linter directly (again, don't commit this within
654 the Nixpkgs repository):
655 </para>
656 <programlisting language="bash">
657 testScript =
658 ''
659 # fmt: off
660 Python code…
661 # fmt: on
662 '';
663</programlisting>
664 <para>
665 Similarly, the type checking of test scripts can be disabled in
666 the following way:
667 </para>
668 <programlisting language="bash">
669{
670 skipTypeCheck = true;
671 nodes.machine =
672 { config, pkgs, ... }:
673 { configuration…
674 };
675}
676</programlisting>
677 </section>
678 <section xml:id="ssec-failing-tests-early">
679 <title>Failing tests early</title>
680 <para>
681 To fail tests early when certain invariants are no longer met
682 (instead of waiting for the build to time out), the decorator
683 <literal>polling_condition</literal> is provided. For example, if
684 we are testing a program <literal>foo</literal> that should not
685 quit after being started, we might write the following:
686 </para>
687 <programlisting language="python">
688@polling_condition
689def foo_running():
690 machine.succeed("pgrep -x foo")
691
692
693machine.succeed("foo --start")
694machine.wait_until_succeeds("pgrep -x foo")
695
696with foo_running:
697 ... # Put `foo` through its paces
698</programlisting>
699 <para>
700 <literal>polling_condition</literal> takes the following
701 (optional) arguments:
702 </para>
703 <para>
704 <literal>seconds_interval</literal>
705 </para>
706 <para>
707 : specifies how often the condition should be polled:
708 </para>
709 <programlisting language="python">
710@polling_condition(seconds_interval=10)
711def foo_running():
712 machine.succeed("pgrep -x foo")
713</programlisting>
714 <para>
715 <literal>description</literal>
716 </para>
717 <para>
718 : is used in the log when the condition is checked. If this is not
719 provided, the description is pulled from the docstring of the
720 function. These two are therefore equivalent:
721 </para>
722 <programlisting language="python">
723@polling_condition
724def foo_running():
725 "check that foo is running"
726 machine.succeed("pgrep -x foo")
727</programlisting>
728 <programlisting language="python">
729@polling_condition(description="check that foo is running")
730def foo_running():
731 machine.succeed("pgrep -x foo")
732</programlisting>
733 </section>
734 <section xml:id="ssec-python-packages-in-test-script">
735 <title>Adding Python packages to the test script</title>
736 <para>
737 When additional Python libraries are required in the test script,
738 they can be added using the parameter
739 <literal>extraPythonPackages</literal>. For example, you could add
740 <literal>numpy</literal> like this:
741 </para>
742 <programlisting language="bash">
743{
744 extraPythonPackages = p: [ p.numpy ];
745
746 nodes = { };
747
748 # Type checking on extra packages doesn't work yet
749 skipTypeCheck = true;
750
751 testScript = ''
752 import numpy as np
753 assert str(np.zeros(4) == "array([0., 0., 0., 0.])")
754 '';
755}
756</programlisting>
757 <para>
758 In that case, <literal>numpy</literal> is chosen from the generic
759 <literal>python3Packages</literal>.
760 </para>
761 </section>
762 <section xml:id="sec-test-options-reference">
763 <title>Test Options Reference</title>
764 <para>
765 The following options can be used when writing tests.
766 </para>
767 <xi:include href="../../generated/test-options-db.xml" xpointer="test-options-list"/>
768 </section>
769</section>