pyrox.dev
1<section xmlns="http://docbook.org/ns/docbook"
2 xmlns:xlink="http://www.w3.org/1999/xlink"
3 xmlns:xi="http://www.w3.org/2001/XInclude"
4 version="5.0"
5 xml:id="sec-writing-nixos-tests">
6
7<title>Writing Tests</title>
8
9<para>A NixOS test is a Nix expression that has the following structure:
10
11<programlisting>
12import ./make-test.nix {
13
14 # Either the configuration of a single machine:
15 machine =
16 { config, pkgs, ... }:
17 { <replaceable>configuration…</replaceable>
18 };
19
20 # Or a set of machines:
21 nodes =
22 { <replaceable>machine1</replaceable> =
23 { config, pkgs, ... }: { <replaceable>…</replaceable> };
24 <replaceable>machine2</replaceable> =
25 { config, pkgs, ... }: { <replaceable>…</replaceable> };
26 …
27 };
28
29 testScript =
30 ''
31 <replaceable>Perl code…</replaceable>
32 '';
33}
34</programlisting>
35
36The attribute <literal>testScript</literal> is a bit of Perl code that
37executes the test (described below). During the test, it will start
38one or more virtual machines, the configuration of which is described
39by the attribute <literal>machine</literal> (if you need only one
40machine in your test) or by the attribute <literal>nodes</literal> (if
41you need multiple machines). For instance, <filename
42xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/login.nix">login.nix</filename>
43only needs a single machine to test whether users can log in on the
44virtual console, whether device ownership is correctly maintained when
45switching between consoles, and so on. On the other hand, <filename
46xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/nfs.nix">nfs.nix</filename>,
47which tests NFS client and server functionality in the Linux kernel
48(including whether locks are maintained across server crashes),
49requires three machines: a server and two clients.</para>
50
51<para>There are a few special NixOS configuration options for test
52VMs:
53
54<!-- FIXME: would be nice to generate this automatically. -->
55
56<variablelist>
57
58 <varlistentry>
59 <term><option>virtualisation.memorySize</option></term>
60 <listitem><para>The memory of the VM in
61 megabytes.</para></listitem>
62 </varlistentry>
63
64 <varlistentry>
65 <term><option>virtualisation.vlans</option></term>
66 <listitem><para>The virtual networks to which the VM is
67 connected. See <filename
68 xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/nat.nix">nat.nix</filename>
69 for an example.</para></listitem>
70 </varlistentry>
71
72 <varlistentry>
73 <term><option>virtualisation.writableStore</option></term>
74 <listitem><para>By default, the Nix store in the VM is not
75 writable. If you enable this option, a writable union file system
76 is mounted on top of the Nix store to make it appear
77 writable. This is necessary for tests that run Nix operations that
78 modify the store.</para></listitem>
79 </varlistentry>
80
81</variablelist>
82
83For more options, see the module <filename
84xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/qemu-vm.nix">qemu-vm.nix</filename>.</para>
85
86<para>The test script is a sequence of Perl statements that perform
87various actions, such as starting VMs, executing commands in the VMs,
88and so on. Each virtual machine is represented as an object stored in
89the variable <literal>$<replaceable>name</replaceable></literal>,
90where <replaceable>name</replaceable> is the identifier of the machine
91(which is just <literal>machine</literal> if you didn’t specify
92multiple machines using the <literal>nodes</literal> attribute). For
93instance, the following starts the machine, waits until it has
94finished booting, then executes a command and checks that the output
95is more-or-less correct:
96
97<programlisting>
98$machine->start;
99$machine->waitForUnit("default.target");
100$machine->succeed("uname") =~ /Linux/;
101</programlisting>
102
103The first line is actually unnecessary; machines are implicitly
104started when you first execute an action on them (such as
105<literal>waitForUnit</literal> or <literal>succeed</literal>). If you
106have multiple machines, you can speed up the test by starting them in
107parallel:
108
109<programlisting>
110startAll;
111</programlisting>
112
113</para>
114
115<para>The following methods are available on machine objects:
116
117<variablelist>
118
119 <varlistentry>
120 <term><methodname>start</methodname></term>
121 <listitem><para>Start the virtual machine. This method is
122 asynchronous — it does not wait for the machine to finish
123 booting.</para></listitem>
124 </varlistentry>
125
126 <varlistentry>
127 <term><methodname>shutdown</methodname></term>
128 <listitem><para>Shut down the machine, waiting for the VM to
129 exit.</para></listitem>
130 </varlistentry>
131
132 <varlistentry>
133 <term><methodname>crash</methodname></term>
134 <listitem><para>Simulate a sudden power failure, by telling the VM
135 to exit immediately.</para></listitem>
136 </varlistentry>
137
138 <varlistentry>
139 <term><methodname>block</methodname></term>
140 <listitem><para>Simulate unplugging the Ethernet cable that
141 connects the machine to the other machines.</para></listitem>
142 </varlistentry>
143
144 <varlistentry>
145 <term><methodname>unblock</methodname></term>
146 <listitem><para>Undo the effect of
147 <methodname>block</methodname>.</para></listitem>
148 </varlistentry>
149
150 <varlistentry>
151 <term><methodname>screenshot</methodname></term>
152 <listitem><para>Take a picture of the display of the virtual
153 machine, in PNG format. The screenshot is linked from the HTML
154 log.</para></listitem>
155 </varlistentry>
156
157 <varlistentry>
158 <term><methodname>getScreenText</methodname></term>
159 <listitem><para>Return a textual representation of what is currently
160 visible on the machine's screen using optical character
161 recognition.</para>
162 <note><para>This requires passing <option>enableOCR</option> to the test
163 attribute set.</para></note></listitem>
164 </varlistentry>
165
166 <varlistentry>
167 <term><methodname>sendMonitorCommand</methodname></term>
168 <listitem><para>Send a command to the QEMU monitor. This is rarely
169 used, but allows doing stuff such as attaching virtual USB disks
170 to a running machine.</para></listitem>
171 </varlistentry>
172
173 <varlistentry>
174 <term><methodname>sendKeys</methodname></term>
175 <listitem><para>Simulate pressing keys on the virtual keyboard,
176 e.g., <literal>sendKeys("ctrl-alt-delete")</literal>.</para></listitem>
177 </varlistentry>
178
179 <varlistentry>
180 <term><methodname>sendChars</methodname></term>
181 <listitem><para>Simulate typing a sequence of characters on the
182 virtual keyboard, e.g., <literal>sendKeys("foobar\n")</literal>
183 will type the string <literal>foobar</literal> followed by the
184 Enter key.</para></listitem>
185 </varlistentry>
186
187 <varlistentry>
188 <term><methodname>execute</methodname></term>
189 <listitem><para>Execute a shell command, returning a list
190 <literal>(<replaceable>status</replaceable>,
191 <replaceable>stdout</replaceable>)</literal>.</para></listitem>
192 </varlistentry>
193
194 <varlistentry>
195 <term><methodname>succeed</methodname></term>
196 <listitem><para>Execute a shell command, raising an exception if
197 the exit status is not zero, otherwise returning the standard
198 output.</para></listitem>
199 </varlistentry>
200
201 <varlistentry>
202 <term><methodname>fail</methodname></term>
203 <listitem><para>Like <methodname>succeed</methodname>, but raising
204 an exception if the command returns a zero status.</para></listitem>
205 </varlistentry>
206
207 <varlistentry>
208 <term><methodname>waitUntilSucceeds</methodname></term>
209 <listitem><para>Repeat a shell command with 1-second intervals
210 until it succeeds.</para></listitem>
211 </varlistentry>
212
213 <varlistentry>
214 <term><methodname>waitUntilFails</methodname></term>
215 <listitem><para>Repeat a shell command with 1-second intervals
216 until it fails.</para></listitem>
217 </varlistentry>
218
219 <varlistentry>
220 <term><methodname>waitForUnit</methodname></term>
221 <listitem><para>Wait until the specified systemd unit has reached
222 the “active” state.</para></listitem>
223 </varlistentry>
224
225 <varlistentry>
226 <term><methodname>waitForFile</methodname></term>
227 <listitem><para>Wait until the specified file
228 exists.</para></listitem>
229 </varlistentry>
230
231 <varlistentry>
232 <term><methodname>waitForOpenPort</methodname></term>
233 <listitem><para>Wait until a process is listening on the given TCP
234 port (on <literal>localhost</literal>, at least).</para></listitem>
235 </varlistentry>
236
237 <varlistentry>
238 <term><methodname>waitForClosedPort</methodname></term>
239 <listitem><para>Wait until nobody is listening on the given TCP
240 port.</para></listitem>
241 </varlistentry>
242
243 <varlistentry>
244 <term><methodname>waitForX</methodname></term>
245 <listitem><para>Wait until the X11 server is accepting
246 connections.</para></listitem>
247 </varlistentry>
248
249 <varlistentry>
250 <term><methodname>waitForText</methodname></term>
251 <listitem><para>Wait until the supplied regular expressions matches
252 the textual contents of the screen by using optical character recognition
253 (see <methodname>getScreenText</methodname>).</para>
254 <note><para>This requires passing <option>enableOCR</option> to the test
255 attribute set.</para></note></listitem>
256 </varlistentry>
257
258 <varlistentry>
259 <term><methodname>waitForWindow</methodname></term>
260 <listitem><para>Wait until an X11 window has appeared whose name
261 matches the given regular expression, e.g.,
262 <literal>waitForWindow(qr/Terminal/)</literal>.</para></listitem>
263 </varlistentry>
264
265 <varlistentry>
266 <term><methodname>copyFileFromHost</methodname></term>
267 <listitem><para>Copies a file from host to machine, e.g.,
268 <literal>copyFileFromHost("myfile", "/etc/my/important/file")</literal>.</para>
269 <para>The first argument is the file on the host. The file needs to be
270 accessible while building the nix derivation. The second argument is
271 the location of the file on the machine.</para>
272 </listitem>
273 </varlistentry>
274
275 <varlistentry>
276 <term><methodname>systemctl</methodname></term>
277 <listitem>
278 <para>Runs <literal>systemctl</literal> commands with optional support for
279 <literal>systemctl --user</literal></para>
280 <para>
281 <programlisting>
282 $machine->systemctl("list-jobs --no-pager"); // runs `systemctl list-jobs --no-pager`
283 $machine->systemctl("list-jobs --no-pager", "any-user"); // spawns a shell for `any-user` and runs `systemctl --user list-jobs --no-pager`
284 </programlisting>
285 </para>
286 </listitem>
287 </varlistentry>
288
289</variablelist>
290
291</para>
292
293<para>
294 To test user units declared by <literal>systemd.user.services</literal> the optional <literal>$user</literal>
295 argument can be used:
296
297 <programlisting>
298 $machine->start;
299 $machine->waitForX;
300 $machine->waitForUnit("xautolock.service", "x-session-user");
301 </programlisting>
302
303 This applies to <literal>systemctl</literal>, <literal>getUnitInfo</literal>,
304 <literal>waitForUnit</literal>, <literal>startJob</literal>
305 and <literal>stopJob</literal>.
306</para>
307
308</section>