pyrox.dev
1<section xmlns="http://docbook.org/ns/docbook"
2 xmlns:xlink="http://www.w3.org/1999/xlink"
3 xmlns:xi="http://www.w3.org/2001/XInclude"
4 version="5.0"
5 xml:id="sec-writing-nixos-tests">
6 <title>Writing Tests</title>
7
8 <para>
9 A NixOS test is a Nix expression that has the following structure:
10<programlisting>
11import ./make-test.nix {
12
13 # Either the configuration of a single machine:
14 machine =
15 { config, pkgs, ... }:
16 { <replaceable>configuration…</replaceable>
17 };
18
19 # Or a set of machines:
20 nodes =
21 { <replaceable>machine1</replaceable> =
22 { config, pkgs, ... }: { <replaceable>…</replaceable> };
23 <replaceable>machine2</replaceable> =
24 { config, pkgs, ... }: { <replaceable>…</replaceable> };
25 …
26 };
27
28 testScript =
29 ''
30 <replaceable>Perl code…</replaceable>
31 '';
32}
33</programlisting>
34 The attribute <literal>testScript</literal> is a bit of Perl code that
35 executes the test (described below). During the test, it will start one or
36 more virtual machines, the configuration of which is described by the
37 attribute <literal>machine</literal> (if you need only one machine in your
38 test) or by the attribute <literal>nodes</literal> (if you need multiple
39 machines). For instance,
40 <filename
41xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/login.nix">login.nix</filename>
42 only needs a single machine to test whether users can log in on the virtual
43 console, whether device ownership is correctly maintained when switching
44 between consoles, and so on. On the other hand,
45 <filename
46xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/nfs.nix">nfs.nix</filename>,
47 which tests NFS client and server functionality in the Linux kernel
48 (including whether locks are maintained across server crashes), requires
49 three machines: a server and two clients.
50 </para>
51
52 <para>
53 There are a few special NixOS configuration options for test VMs:
54<!-- FIXME: would be nice to generate this automatically. -->
55 <variablelist>
56 <varlistentry>
57 <term>
58 <option>virtualisation.memorySize</option>
59 </term>
60 <listitem>
61 <para>
62 The memory of the VM in megabytes.
63 </para>
64 </listitem>
65 </varlistentry>
66 <varlistentry>
67 <term>
68 <option>virtualisation.vlans</option>
69 </term>
70 <listitem>
71 <para>
72 The virtual networks to which the VM is connected. See
73 <filename
74 xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/nat.nix">nat.nix</filename>
75 for an example.
76 </para>
77 </listitem>
78 </varlistentry>
79 <varlistentry>
80 <term>
81 <option>virtualisation.writableStore</option>
82 </term>
83 <listitem>
84 <para>
85 By default, the Nix store in the VM is not writable. If you enable this
86 option, a writable union file system is mounted on top of the Nix store
87 to make it appear writable. This is necessary for tests that run Nix
88 operations that modify the store.
89 </para>
90 </listitem>
91 </varlistentry>
92 </variablelist>
93 For more options, see the module
94 <filename
95xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/qemu-vm.nix">qemu-vm.nix</filename>.
96 </para>
97
98 <para>
99 The test script is a sequence of Perl statements that perform various
100 actions, such as starting VMs, executing commands in the VMs, and so on. Each
101 virtual machine is represented as an object stored in the variable
102 <literal>$<replaceable>name</replaceable></literal>, where
103 <replaceable>name</replaceable> is the identifier of the machine (which is
104 just <literal>machine</literal> if you didn’t specify multiple machines
105 using the <literal>nodes</literal> attribute). For instance, the following
106 starts the machine, waits until it has finished booting, then executes a
107 command and checks that the output is more-or-less correct:
108<programlisting>
109$machine->start;
110$machine->waitForUnit("default.target");
111$machine->succeed("uname") =~ /Linux/;
112</programlisting>
113 The first line is actually unnecessary; machines are implicitly started when
114 you first execute an action on them (such as <literal>waitForUnit</literal>
115 or <literal>succeed</literal>). If you have multiple machines, you can speed
116 up the test by starting them in parallel:
117<programlisting>
118startAll;
119</programlisting>
120 </para>
121
122 <para>
123 The following methods are available on machine objects:
124 <variablelist>
125 <varlistentry>
126 <term>
127 <methodname>start</methodname>
128 </term>
129 <listitem>
130 <para>
131 Start the virtual machine. This method is asynchronous — it does not
132 wait for the machine to finish booting.
133 </para>
134 </listitem>
135 </varlistentry>
136 <varlistentry>
137 <term>
138 <methodname>shutdown</methodname>
139 </term>
140 <listitem>
141 <para>
142 Shut down the machine, waiting for the VM to exit.
143 </para>
144 </listitem>
145 </varlistentry>
146 <varlistentry>
147 <term>
148 <methodname>crash</methodname>
149 </term>
150 <listitem>
151 <para>
152 Simulate a sudden power failure, by telling the VM to exit immediately.
153 </para>
154 </listitem>
155 </varlistentry>
156 <varlistentry>
157 <term>
158 <methodname>block</methodname>
159 </term>
160 <listitem>
161 <para>
162 Simulate unplugging the Ethernet cable that connects the machine to the
163 other machines.
164 </para>
165 </listitem>
166 </varlistentry>
167 <varlistentry>
168 <term>
169 <methodname>unblock</methodname>
170 </term>
171 <listitem>
172 <para>
173 Undo the effect of <methodname>block</methodname>.
174 </para>
175 </listitem>
176 </varlistentry>
177 <varlistentry>
178 <term>
179 <methodname>screenshot</methodname>
180 </term>
181 <listitem>
182 <para>
183 Take a picture of the display of the virtual machine, in PNG format. The
184 screenshot is linked from the HTML log.
185 </para>
186 </listitem>
187 </varlistentry>
188 <varlistentry>
189 <term>
190 <methodname>getScreenText</methodname>
191 </term>
192 <listitem>
193 <para>
194 Return a textual representation of what is currently visible on the
195 machine's screen using optical character recognition.
196 </para>
197 <note>
198 <para>
199 This requires passing <option>enableOCR</option> to the test attribute
200 set.
201 </para>
202 </note>
203 </listitem>
204 </varlistentry>
205 <varlistentry>
206 <term>
207 <methodname>sendMonitorCommand</methodname>
208 </term>
209 <listitem>
210 <para>
211 Send a command to the QEMU monitor. This is rarely used, but allows doing
212 stuff such as attaching virtual USB disks to a running machine.
213 </para>
214 </listitem>
215 </varlistentry>
216 <varlistentry>
217 <term>
218 <methodname>sendKeys</methodname>
219 </term>
220 <listitem>
221 <para>
222 Simulate pressing keys on the virtual keyboard, e.g.,
223 <literal>sendKeys("ctrl-alt-delete")</literal>.
224 </para>
225 </listitem>
226 </varlistentry>
227 <varlistentry>
228 <term>
229 <methodname>sendChars</methodname>
230 </term>
231 <listitem>
232 <para>
233 Simulate typing a sequence of characters on the virtual keyboard, e.g.,
234 <literal>sendKeys("foobar\n")</literal> will type the string
235 <literal>foobar</literal> followed by the Enter key.
236 </para>
237 </listitem>
238 </varlistentry>
239 <varlistentry>
240 <term>
241 <methodname>execute</methodname>
242 </term>
243 <listitem>
244 <para>
245 Execute a shell command, returning a list
246 <literal>(<replaceable>status</replaceable>,
247 <replaceable>stdout</replaceable>)</literal>.
248 </para>
249 </listitem>
250 </varlistentry>
251 <varlistentry>
252 <term>
253 <methodname>succeed</methodname>
254 </term>
255 <listitem>
256 <para>
257 Execute a shell command, raising an exception if the exit status is not
258 zero, otherwise returning the standard output.
259 </para>
260 </listitem>
261 </varlistentry>
262 <varlistentry>
263 <term>
264 <methodname>fail</methodname>
265 </term>
266 <listitem>
267 <para>
268 Like <methodname>succeed</methodname>, but raising an exception if the
269 command returns a zero status.
270 </para>
271 </listitem>
272 </varlistentry>
273 <varlistentry>
274 <term>
275 <methodname>waitUntilSucceeds</methodname>
276 </term>
277 <listitem>
278 <para>
279 Repeat a shell command with 1-second intervals until it succeeds.
280 </para>
281 </listitem>
282 </varlistentry>
283 <varlistentry>
284 <term>
285 <methodname>waitUntilFails</methodname>
286 </term>
287 <listitem>
288 <para>
289 Repeat a shell command with 1-second intervals until it fails.
290 </para>
291 </listitem>
292 </varlistentry>
293 <varlistentry>
294 <term>
295 <methodname>waitForUnit</methodname>
296 </term>
297 <listitem>
298 <para>
299 Wait until the specified systemd unit has reached the “active” state.
300 </para>
301 </listitem>
302 </varlistentry>
303 <varlistentry>
304 <term>
305 <methodname>waitForFile</methodname>
306 </term>
307 <listitem>
308 <para>
309 Wait until the specified file exists.
310 </para>
311 </listitem>
312 </varlistentry>
313 <varlistentry>
314 <term>
315 <methodname>waitForOpenPort</methodname>
316 </term>
317 <listitem>
318 <para>
319 Wait until a process is listening on the given TCP port (on
320 <literal>localhost</literal>, at least).
321 </para>
322 </listitem>
323 </varlistentry>
324 <varlistentry>
325 <term>
326 <methodname>waitForClosedPort</methodname>
327 </term>
328 <listitem>
329 <para>
330 Wait until nobody is listening on the given TCP port.
331 </para>
332 </listitem>
333 </varlistentry>
334 <varlistentry>
335 <term>
336 <methodname>waitForX</methodname>
337 </term>
338 <listitem>
339 <para>
340 Wait until the X11 server is accepting connections.
341 </para>
342 </listitem>
343 </varlistentry>
344 <varlistentry>
345 <term>
346 <methodname>waitForText</methodname>
347 </term>
348 <listitem>
349 <para>
350 Wait until the supplied regular expressions matches the textual contents
351 of the screen by using optical character recognition (see
352 <methodname>getScreenText</methodname>).
353 </para>
354 <note>
355 <para>
356 This requires passing <option>enableOCR</option> to the test attribute
357 set.
358 </para>
359 </note>
360 </listitem>
361 </varlistentry>
362 <varlistentry>
363 <term>
364 <methodname>waitForWindow</methodname>
365 </term>
366 <listitem>
367 <para>
368 Wait until an X11 window has appeared whose name matches the given
369 regular expression, e.g., <literal>waitForWindow(qr/Terminal/)</literal>.
370 </para>
371 </listitem>
372 </varlistentry>
373 <varlistentry>
374 <term>
375 <methodname>copyFileFromHost</methodname>
376 </term>
377 <listitem>
378 <para>
379 Copies a file from host to machine, e.g.,
380 <literal>copyFileFromHost("myfile", "/etc/my/important/file")</literal>.
381 </para>
382 <para>
383 The first argument is the file on the host. The file needs to be
384 accessible while building the nix derivation. The second argument is the
385 location of the file on the machine.
386 </para>
387 </listitem>
388 </varlistentry>
389 <varlistentry>
390 <term>
391 <methodname>systemctl</methodname>
392 </term>
393 <listitem>
394 <para>
395 Runs <literal>systemctl</literal> commands with optional support for
396 <literal>systemctl --user</literal>
397 </para>
398 <para>
399<programlisting>
400 $machine->systemctl("list-jobs --no-pager"); // runs `systemctl list-jobs --no-pager`
401 $machine->systemctl("list-jobs --no-pager", "any-user"); // spawns a shell for `any-user` and runs `systemctl --user list-jobs --no-pager`
402 </programlisting>
403 </para>
404 </listitem>
405 </varlistentry>
406 </variablelist>
407 </para>
408
409 <para>
410 To test user units declared by <literal>systemd.user.services</literal> the
411 optional <literal>$user</literal> argument can be used:
412<programlisting>
413 $machine->start;
414 $machine->waitForX;
415 $machine->waitForUnit("xautolock.service", "x-session-user");
416 </programlisting>
417 This applies to <literal>systemctl</literal>, <literal>getUnitInfo</literal>,
418 <literal>waitForUnit</literal>, <literal>startJob</literal> and
419 <literal>stopJob</literal>.
420 </para>
421</section>