pyrox.dev
1# Testers {#chap-testers}
2
3This chapter describes several testing builders which are available in the `testers` namespace.
4
5## `hasPkgConfigModules` {#tester-hasPkgConfigModules}
6
7<!-- Old anchor name so links still work -->
8[]{#tester-hasPkgConfigModule}
9Checks whether a package exposes a given list of `pkg-config` modules.
10If the `moduleNames` argument is omitted, `hasPkgConfigModules` will use `meta.pkgConfigModules`.
11
12:::{.example #ex-haspkgconfigmodules-defaultvalues}
13
14# Check that `pkg-config` modules are exposed using default values
15
16```nix
17{
18 passthru.tests.pkg-config = testers.hasPkgConfigModules {
19 package = finalAttrs.finalPackage;
20 };
21
22 meta.pkgConfigModules = [ "libfoo" ];
23}
24```
25
26:::
27
28:::{.example #ex-haspkgconfigmodules-explicitmodules}
29
30# Check that `pkg-config` modules are exposed using explicit module names
31
32```nix
33{
34 passthru.tests.pkg-config = testers.hasPkgConfigModules {
35 package = finalAttrs.finalPackage;
36 moduleNames = [ "libfoo" ];
37 };
38}
39```
40
41:::
42
43## `lycheeLinkCheck` {#tester-lycheeLinkCheck}
44
45Check a packaged static site's links with the [`lychee` package](https://search.nixos.org/packages?show=lychee&type=packages&query=lychee).
46
47You may use Nix to reproducibly build static websites, such as for software documentation.
48Some packages will install documentation in their `out` or `doc` outputs, or maybe you have dedicated package where you've made your static site reproducible by running a generator, such as [Hugo](https://gohugo.io/) or [mdBook](https://rust-lang.github.io/mdBook/), in a derivation.
49
50If you have a static site that can be built with Nix, you can use `lycheeLinkCheck` to check that the hyperlinks in your site are correct, and do so as part of your Nix workflow and CI.
51
52:::{.example #ex-lycheelinkcheck}
53
54# Check hyperlinks in the `nix` documentation
55
56```nix
57testers.lycheeLinkCheck {
58 site = nix.doc + "/share/doc/nix/manual";
59}
60```
61
62:::
63
64### Return value {#tester-lycheeLinkCheck-return}
65
66This tester produces a package that does not produce useful outputs, but only succeeds if the hyperlinks in your site are correct. The build log will list the broken links.
67
68It has two modes:
69
70- Build the returned derivation; its build process will check that internal hyperlinks are correct. This runs in the sandbox, so it will not check external hyperlinks, but it is quick and reliable.
71
72- Invoke the `.online` attribute with [`nix run`](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-run) ([experimental](https://nixos.org/manual/nix/stable/contributing/experimental-features#xp-feature-nix-command)). This runs outside the sandbox, and checks that both internal and external hyperlinks are correct.
73 Example:
74
75 ```shell
76 nix run nixpkgs#lychee.tests.ok.online
77 ```
78
79### Inputs {#tester-lycheeLinkCheck-inputs}
80
81`site` (path or derivation) {#tester-lycheeLinkCheck-param-site}
82
83: The path to the files to check.
84
85`remap` (attribe set, optional) {#tester-lycheeLinkCheck-param-remap}
86
87: An attribute set where the attribute names are regular expressions.
88 The values should be strings, derivations, or path values.
89
90 In the returned check's default configuration, external URLs are only checked when you run the `.online` attribute.
91
92 By adding remappings, you can check offline that URLs to external resources are correct, by providing a stand-in from the file system.
93
94 Before checking the existence of a URL, the regular expressions are matched and replaced by their corresponding values.
95
96 Example:
97
98 ```nix
99 {
100 "https://nix\\.dev/manual/nix/[a-z0-9.-]*" = "${nix.doc}/share/doc/nix/manual";
101 "https://nixos\\.org/manual/nix/(un)?stable" =
102 "${emptyDirectory}/placeholder-to-disallow-old-nix-docs-urls";
103 }
104 ```
105
106 Store paths in the attribute values are automatically prefixed with `file://`, because lychee requires this for paths in the file system.
107 If this is a problem, or if you need to control the order in which replacements are performed, use `extraConfig.remap` instead.
108
109`extraConfig` (attribute set) {#tester-lycheeLinkCheck-param-extraConfig}
110
111: Extra configuration to pass to `lychee` in its [configuration file](https://github.com/lycheeverse/lychee/blob/master/lychee.example.toml).
112 It is automatically [translated](https://nixos.org/manual/nixos/stable/index.html#sec-settings-nix-representable) to TOML.
113
114 Example: `{ "include_verbatim" = true; }`
115
116`lychee` (derivation, optional) {#tester-lycheeLinkCheck-param-lychee}
117
118: The `lychee` package to use.
119
120## `shellcheck` {#tester-shellcheck}
121
122Run files through `shellcheck`, a static analysis tool for shell scripts, failing if there are any issues.
123
124:::{.example #ex-shellcheck}
125# Run `testers.shellcheck`
126
127A single script
128
129```nix
130testers.shellcheck {
131 name = "script";
132 src = ./script.sh;
133}
134```
135
136Multiple files
137
138```nix
139let
140 inherit (lib) fileset;
141in
142testers.shellcheck {
143 name = "nixbsd-activate";
144 src = fileset.toSource {
145 root = ./.;
146 fileset = fileset.unions [
147 ./lib.sh
148 ./nixbsd-activate
149 ];
150 };
151}
152```
153
154:::
155
156### Inputs {#tester-shellcheck-inputs}
157
158`name` (string, optional)
159: The name of the test.
160 `name` will be required at a future point because it massively improves traceability of test failures, but is kept optional for now to avoid breaking existing usages.
161 Defaults to `run-shellcheck`.
162 The name of the derivation produced by the tester is `shellcheck-${name}` when `name` is supplied.
163
164`src` (path-like)
165: The path to the shell script(s) to check.
166 This can be a single file or a directory containing shell files.
167 All files in `src` will be checked, so you may want to provide `fileset`-based source instead of a whole directory.
168
169### Return value {#tester-shellcheck-return}
170
171A derivation that runs `shellcheck` on the given script(s), producing an empty output if no issues are found.
172The build will fail if `shellcheck` finds any issues.
173
174## `shfmt` {#tester-shfmt}
175
176Run files through `shfmt`, a shell script formatter, failing if any files are reformatted.
177
178:::{.example #ex-shfmt}
179# Run `testers.shfmt`
180
181A single script
182
183```nix
184testers.shfmt {
185 name = "script";
186 src = ./script.sh;
187}
188```
189
190Multiple files
191
192```nix
193let
194 inherit (lib) fileset;
195in
196testers.shfmt {
197 name = "nixbsd";
198 src = fileset.toSource {
199 root = ./.;
200 fileset = fileset.unions [
201 ./lib.sh
202 ./nixbsd-activate
203 ];
204 };
205}
206```
207
208:::
209
210### Inputs {#tester-shfmt-inputs}
211
212`name` (string)
213: The name of the test.
214 `name` is required because it massively improves traceability of test failures.
215 The name of the derivation produced by the tester is `shfmt-${name}`.
216
217`src` (path-like)
218: The path to the shell script(s) to check.
219 This can be a single file or a directory containing shell files.
220 All files in `src` will be checked, so you may want to provide `fileset`-based source instead of a whole directory.
221
222`indent` (integer, optional)
223: The number of spaces to use for indentation.
224 Defaults to `2`.
225 A value of `0` indents with tabs.
226
227### Return value {#tester-shfmt-return}
228
229A derivation that runs `shfmt` on the given script(s), producing an empty output upon success.
230The build will fail if `shfmt` reformats anything.
231
232## `testVersion` {#tester-testVersion}
233
234Checks that the output from running a command contains the specified version string in it as a whole word.
235
236NOTE: This is a check you add to `passthru.tests` which is mainly run by OfBorg, but not in Hydra. If you want a version check failure to block the build altogether, then [`versionCheckHook`](#versioncheckhook) is the tool you're looking for (and recommended for quick builds). The motivation for adding either of these checks would be:
237
238- Catch dynamic linking errors and such and missing environment variables that should be added by wrapping.
239- Probable protection against accidentally building the wrong version, for example when using an "old" hash in a fixed-output derivation.
240
241By default, the command to be run will be inferred from the given `package` attribute:
242it will check `meta.mainProgram` first, and fall back to `pname` or `name`.
243The default argument to the command is `--version`, and the version to be checked will be inferred from the given `package` attribute as well.
244
245:::{.example #ex-testversion-hello}
246
247# Check a program version using all the default values
248
249This example will run the command `hello --version`, and then check that the version of the `hello` package is in the output of the command.
250
251```nix
252{
253 passthru.tests.version = testers.testVersion { package = hello; };
254}
255```
256
257:::
258
259:::{.example #ex-testversion-different-commandversion}
260
261# Check the program version using a specified command and expected version string
262
263This example will run the command `leetcode -V`, and then check that `leetcode 0.4.2` is in the output of the command as a whole word (separated by whitespaces).
264This means that an output like "leetcode 0.4.21" would fail the tests, and an output like "You're running leetcode 0.4.2" would pass the tests.
265
266A common usage of the `version` attribute is to specify `version = "v${version}"`.
267
268```nix
269{
270 version = "0.4.2";
271
272 passthru.tests.version = testers.testVersion {
273 package = leetcode-cli;
274 command = "leetcode -V";
275 version = "leetcode ${version}";
276 };
277}
278```
279
280:::
281
282## `testBuildFailure` {#tester-testBuildFailure}
283
284Make sure that a build does not succeed. This is useful for testing testers.
285
286This returns a derivation with an override on the builder, with the following effects:
287
288 - Fail the build when the original builder succeeds
289 - Move `$out` to `$out/result`, if it exists (assuming `out` is the default output)
290 - Save the build log to `$out/testBuildFailure.log` (same)
291
292While `testBuildFailure` is designed to keep changes to the original builder's environment to a minimum, some small changes are inevitable:
293
294 - The file `$TMPDIR/testBuildFailure.log` is present. It should not be deleted.
295 - `stdout` and `stderr` are a pipe instead of a tty. This could be improved.
296 - One or two extra processes are present in the sandbox during the original builder's execution.
297 - The derivation and output hashes are different, but not unusual.
298 - The derivation includes a dependency on `buildPackages.bash` and `expect-failure.sh`, which is built to include a transitive dependency on `buildPackages.coreutils` and possibly more.
299 These are not added to `PATH` or any other environment variable, so they should be hard to observe.
300
301:::{.example #ex-testBuildFailure-showingenvironmentchanges}
302
303# Check that a build fails, and verify the changes made during build
304
305```nix
306runCommand "example"
307 {
308 failed = testers.testBuildFailure (
309 runCommand "fail" { } ''
310 echo ok-ish >$out
311 echo failing though
312 exit 3
313 ''
314 );
315 }
316 ''
317 grep -F 'ok-ish' $failed/result
318 grep -F 'failing though' $failed/testBuildFailure.log
319 [[ 3 = $(cat $failed/testBuildFailure.exit) ]]
320 touch $out
321 ''
322```
323
324:::
325
326## `testBuildFailure'` {#tester-testBuildFailurePrime}
327
328This tester wraps the functionality provided by [`testers.testBuildFailure`](#tester-testBuildFailure) to make writing checks easier by simplifying checking the exit code of the builder and asserting the existence of entries in the builder's log.
329Additionally, users may specify a script containing additional checks, accessing the result of applying `testers.testBuildFailure` through the variable `failed`.
330
331NOTE: This tester will produce an empty output and exit with success if none of the checks fail; there is no need to `touch "$out"` in `script`.
332
333:::{.example #ex-testBuildFailurePrime-doc-example}
334
335# Check that a build fails, and verify the changes made during build
336
337Re-using the example from [`testers.testBuildFailure`](#ex-testBuildFailure-showingenvironmentchanges), we can see how common checks are made easier and remove the need for `runCommand`:
338
339```nix
340testers.testBuildFailure' {
341 drv = runCommand "doc-example" { } ''
342 echo ok-ish >"$out"
343 echo failing though
344 exit 3
345 '';
346 expectedBuilderExitCode = 3;
347 expectedBuilderLogEntries = [ "failing though" ];
348 script = ''
349 grep --silent -F 'ok-ish' "$failed/result"
350 '';
351}
352```
353
354:::
355
356### Inputs {#tester-testBuildFailurePrime-inputs}
357
358`drv` (derivation)
359
360: The failing derivation to wrap with `testBuildFailure`.
361
362`name` (string, optional)
363
364: The name of the test.
365 When not provided, this value defaults to `testBuildFailure-${(testers.testBuildFailure drv).name}`.
366
367`expectedBuilderExitCode` (integer, optional)
368
369: The expected exit code of the builder of `drv`.
370 When not provided, this value defaults to `1`.
371
372`expectedBuilderLogEntries` (array of string-like values, optional)
373
374: A list of string-like values which must be found in the builder's log by exact match.
375 When not provided, this value defaults to `[ ]`.
376
377 NOTE: Patterns and regular expressions are not supported.
378
379`script` (string, optional)
380
381: A string containing additional checks to run.
382 When not provided, this value defaults to `""`.
383 The result of `testers.testBuildFailure drv` is available through the variable `failed`.
384 As an example, the builder's log is at `"$failed/testBuildFailure.log"`.
385
386### Return value {#tester-testBuildFailurePrime-return}
387
388The tester produces an empty output and only succeeds when the checks using `expectedBuilderExitCode`, `expectedBuilderLogEntries`, and `script` succeed.
389
390## `testEqualContents` {#tester-testEqualContents}
391
392Check that two paths have the same contents.
393
394:::{.example #ex-testEqualContents-toyexample}
395
396# Check that two paths have the same contents
397
398```nix
399testers.testEqualContents {
400 assertion = "sed -e performs replacement";
401 expected = writeText "expected" ''
402 foo baz baz
403 '';
404 actual =
405 runCommand "actual"
406 {
407 # not really necessary for a package that's in stdenv
408 nativeBuildInputs = [ gnused ];
409 base = writeText "base" ''
410 foo bar baz
411 '';
412 }
413 ''
414 sed -e 's/bar/baz/g' $base >$out
415 '';
416}
417```
418
419:::
420
421## `testEqualArrayOrMap` {#tester-testEqualArrayOrMap}
422
423Check that bash arrays (including associative arrays, referred to as "maps") are populated correctly.
424
425This can be used to ensure setup hooks are registered in a certain order, or to write unit tests for shell functions which transform arrays.
426
427:::{.example #ex-testEqualArrayOrMap-test-function-add-cowbell}
428
429# Test a function which appends a value to an array
430
431```nix
432testers.testEqualArrayOrMap {
433 name = "test-function-add-cowbell";
434 valuesArray = [
435 "cowbell"
436 "cowbell"
437 ];
438 expectedArray = [
439 "cowbell"
440 "cowbell"
441 "cowbell"
442 ];
443 script = ''
444 addCowbell() {
445 local -rn arrayNameRef="$1"
446 arrayNameRef+=( "cowbell" )
447 }
448
449 nixLog "appending all values in valuesArray to actualArray"
450 for value in "''${valuesArray[@]}"; do
451 actualArray+=( "$value" )
452 done
453
454 nixLog "applying addCowbell"
455 addCowbell actualArray
456 '';
457}
458```
459
460:::
461
462### Inputs {#tester-testEqualArrayOrMap-inputs}
463
464NOTE: Internally, this tester uses `__structuredAttrs` to handle marshalling between Nix expressions and shell variables.
465This imposes the restriction that arrays and "maps" have values which are string-like.
466
467NOTE: At least one of `expectedArray` and `expectedMap` must be provided.
468
469`name` (string)
470
471: The name of the test.
472
473`script` (string)
474
475: The singular task of `script` is to populate `actualArray` or `actualMap` (it may populate both).
476 To do this, `script` may access the following shell variables:
477
478 - `valuesArray` (available when `valuesArray` is provided to the tester)
479 - `valuesMap` (available when `valuesMap` is provided to the tester)
480 - `actualArray` (available when `expectedArray` is provided to the tester)
481 - `actualMap` (available when `expectedMap` is provided to the tester)
482
483 While both `expectedArray` and `expectedMap` are in scope during the execution of `script`, they *must not* be accessed or modified from within `script`.
484
485`valuesArray` (array of string-like values, optional)
486
487: An array of string-like values.
488 This array may be used within `script`.
489
490`valuesMap` (attribute set of string-like values, optional)
491
492: An attribute set of string-like values.
493 This attribute set may be used within `script`.
494
495`expectedArray` (array of string-like values, optional)
496
497: An array of string-like values.
498 This array *must not* be accessed or modified from within `script`.
499 When provided, `script` is expected to populate `actualArray`.
500
501`expectedMap` (attribute set of string-like values, optional)
502
503: An attribute set of string-like values.
504 This attribute set *must not* be accessed or modified from within `script`.
505 When provided, `script` is expected to populate `actualMap`.
506
507### Return value {#tester-testEqualArrayOrMap-return}
508
509The tester produces an empty output and only succeeds when `expectedArray` and `expectedMap` match `actualArray` and `actualMap`, respectively, when non-null.
510The build log will contain differences encountered.
511
512## `testEqualDerivation` {#tester-testEqualDerivation}
513
514Checks that two packages produce the exact same build instructions.
515
516This can be used to make sure that a certain difference of configuration, such as the presence of an overlay does not cause a cache miss.
517
518When the derivations are equal, the return value is an empty file.
519Otherwise, the build log explains the difference via `nix-diff`.
520
521:::{.example #ex-testEqualDerivation-hello}
522
523# Check that two packages produce the same derivation
524
525```nix
526testers.testEqualDerivation "The hello package must stay the same when enabling checks." hello (
527 hello.overrideAttrs (o: {
528 doCheck = true;
529 })
530)
531```
532
533:::
534
535## `invalidateFetcherByDrvHash` {#tester-invalidateFetcherByDrvHash}
536
537Use the derivation hash to invalidate the output via name, for testing.
538
539Type: `(a@{ name, ... } -> Derivation) -> a -> Derivation`
540
541Normally, fixed output derivations can and should be cached by their output hash only, but for testing we want to re-fetch everytime the fetcher changes.
542
543Changes to the fetcher become apparent in the drvPath, which is a hash of how to fetch, rather than a fixed store path.
544By inserting this hash into the name, we can make sure to re-run the fetcher every time the fetcher changes.
545
546This relies on the assumption that Nix isn't clever enough to reuse its database of local store contents to optimize fetching.
547
548You might notice that the "salted" name derives from the normal invocation, not the final derivation.
549`invalidateFetcherByDrvHash` has to invoke the fetcher function twice:
550once to get a derivation hash, and again to produce the final fixed output derivation.
551
552:::{.example #ex-invalidateFetcherByDrvHash-nix}
553
554# Prevent nix from reusing the output of a fetcher
555
556```nix
557{
558 tests.fetchgit = testers.invalidateFetcherByDrvHash fetchgit {
559 name = "nix-source";
560 url = "https://github.com/NixOS/nix";
561 rev = "9d9dbe6ed05854e03811c361a3380e09183f4f4a";
562 hash = "sha256-7DszvbCNTjpzGRmpIVAWXk20P0/XTrWZ79KSOGLrUWY=";
563 };
564}
565```
566
567:::
568
569## `runCommand` {#tester-runCommand}
570
571`runCommand :: { name, script, stdenv ? stdenvNoCC, hash ? "...", ... } -> Derivation`
572
573This is a wrapper around `pkgs.runCommandWith`, which
574- produces a fixed-output derivation, enabling the command(s) to access the network ;
575- salts the derivation's name based on its inputs, ensuring the command is re-run whenever the inputs changes.
576
577It accepts the following attributes:
578- the derivation's `name` ;
579- the `script` to be executed ;
580- `stdenv`, the environment to use, defaulting to `stdenvNoCC` ;
581- the derivation's output `hash`, defaulting to the empty file's.
582 The derivation's `outputHashMode` is set by default to recursive, so the `script` can output a directory as well.
583
584All other attributes are passed through to [`mkDerivation`](#sec-using-stdenv),
585including `nativeBuildInputs` to specify dependencies available to the `script`.
586
587:::{.example #ex-tester-runCommand-nix}
588
589# Run a command with network access
590
591```nix
592testers.runCommand {
593 name = "access-the-internet";
594 script = ''
595 curl -o /dev/null https://example.com
596 touch $out
597 '';
598 nativeBuildInputs = with pkgs; [
599 cacert
600 curl
601 ];
602}
603```
604
605:::
606
607## `runNixOSTest` {#tester-runNixOSTest}
608
609A helper function that behaves exactly like the NixOS `runTest`, except it also assigns this Nixpkgs package set as the `pkgs` of the test and makes the `nixpkgs.*` options read-only.
610
611If your test is part of the Nixpkgs repository, or if you need a more general entrypoint, see ["Calling a test" in the NixOS manual](https://nixos.org/manual/nixos/stable/index.html#sec-calling-nixos-tests).
612
613:::{.example #ex-runNixOSTest-hello}
614
615# Run a NixOS test using `runNixOSTest`
616
617```nix
618pkgs.testers.runNixOSTest (
619 { lib, ... }:
620 {
621 name = "hello";
622 nodes.machine =
623 { pkgs, ... }:
624 {
625 environment.systemPackages = [ pkgs.hello ];
626 };
627 testScript = ''
628 machine.succeed("hello")
629 '';
630 }
631)
632```
633
634:::
635
636## `nixosTest` {#tester-nixosTest}
637
638Run a NixOS VM network test using this evaluation of Nixpkgs.
639
640NOTE: This function is primarily for external use. NixOS itself uses `make-test-python.nix` directly. Packages defined in Nixpkgs [reuse NixOS tests via `nixosTests`, plural](#ssec-nixos-tests-linking).
641
642It is mostly equivalent to the function `import ./make-test-python.nix` from the [NixOS manual](https://nixos.org/nixos/manual/index.html#sec-nixos-tests), except that the current application of Nixpkgs (`pkgs`) will be used, instead of letting NixOS invoke Nixpkgs anew.
643
644If a test machine needs to set NixOS options under `nixpkgs`, it must set only the `nixpkgs.pkgs` option.
645
646### Parameter {#tester-nixosTest-parameter}
647
648A [NixOS VM test network](https://nixos.org/nixos/manual/index.html#sec-nixos-tests), or path to it. Example:
649
650```nix
651{
652 name = "my-test";
653 nodes = {
654 machine1 =
655 {
656 lib,
657 pkgs,
658 nodes,
659 ...
660 }:
661 {
662 environment.systemPackages = [ pkgs.hello ];
663 services.foo.enable = true;
664 };
665 # machine2 = ...;
666 };
667 testScript = ''
668 start_all()
669 machine1.wait_for_unit("foo.service")
670 machine1.succeed("hello | foo-send")
671 '';
672}
673```
674
675### Result {#tester-nixosTest-result}
676
677A derivation that runs the VM test.
678
679Notable attributes:
680
681 * `nodes`: the evaluated NixOS configurations. Useful for debugging and exploring the configuration.
682
683 * `driverInteractive`: a script that launches an interactive Python session in the context of the `testScript`.