pyrox.dev
1# Testers {#chap-testers}
2
3This chapter describes several testing builders which are available in the `testers` namespace.
4
5## `hasPkgConfigModules` {#tester-hasPkgConfigModules}
6
7<!-- Old anchor name so links still work -->
8[]{#tester-hasPkgConfigModule}
9Checks whether a package exposes a given list of `pkg-config` modules.
10If the `moduleNames` argument is omitted, `hasPkgConfigModules` will use `meta.pkgConfigModules`.
11
12:::{.example #ex-haspkgconfigmodules-defaultvalues}
13
14# Check that `pkg-config` modules are exposed using default values
15
16```nix
17{
18 passthru.tests.pkg-config = testers.hasPkgConfigModules { package = finalAttrs.finalPackage; };
19
20 meta.pkgConfigModules = [ "libfoo" ];
21}
22```
23
24:::
25
26:::{.example #ex-haspkgconfigmodules-explicitmodules}
27
28# Check that `pkg-config` modules are exposed using explicit module names
29
30```nix
31{
32 passthru.tests.pkg-config = testers.hasPkgConfigModules {
33 package = finalAttrs.finalPackage;
34 moduleNames = [ "libfoo" ];
35 };
36}
37```
38
39:::
40
41## `hasCmakeConfigModules` {#tester-hasCmakeConfigModules}
42
43Checks whether a package exposes a given list of `*config.cmake` modules.
44Note the moduleNames used in cmake find_package are case sensitive.
45
46:::{.example #ex-hascmakeconfigmodules}
47
48# Check that `*config.cmake` modules are exposed using explicit module names
49
50```nix
51{
52 passthru.tests.cmake-config = testers.hasCmakeConfigModules {
53 package = finalAttrs.finalPackage;
54 moduleNames = [ "Foo" ];
55 };
56}
57```
58
59:::
60
61## `lycheeLinkCheck` {#tester-lycheeLinkCheck}
62
63Check a packaged static site's links with the [`lychee` package](https://search.nixos.org/packages?show=lychee&type=packages&query=lychee).
64
65You may use Nix to reproducibly build static websites, such as for software documentation.
66Some packages will install documentation in their `out` or `doc` outputs, or maybe you have dedicated package where you've made your static site reproducible by running a generator, such as [Hugo](https://gohugo.io/) or [mdBook](https://rust-lang.github.io/mdBook/), in a derivation.
67
68If you have a static site that can be built with Nix, you can use `lycheeLinkCheck` to check that the hyperlinks in your site are correct, and do so as part of your Nix workflow and CI.
69
70:::{.example #ex-lycheelinkcheck}
71
72# Check hyperlinks in the `nix` documentation
73
74```nix
75testers.lycheeLinkCheck { site = nix.doc + "/share/doc/nix/manual"; }
76```
77
78:::
79
80### Return value {#tester-lycheeLinkCheck-return}
81
82This tester produces a package that does not produce useful outputs, but only succeeds if the hyperlinks in your site are correct. The build log will list the broken links.
83
84It has two modes:
85
86- Build the returned derivation; its build process will check that internal hyperlinks are correct. This runs in the sandbox, so it will not check external hyperlinks, but it is quick and reliable.
87
88- Invoke the `.online` attribute with [`nix run`](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-run) ([experimental](https://nixos.org/manual/nix/stable/contributing/experimental-features#xp-feature-nix-command)). This runs outside the sandbox, and checks that both internal and external hyperlinks are correct.
89 Example:
90
91 ```shell
92 nix run nixpkgs#lychee.tests.ok.online
93 ```
94
95### Inputs {#tester-lycheeLinkCheck-inputs}
96
97`site` (path or derivation) {#tester-lycheeLinkCheck-param-site}
98
99: The path to the files to check.
100
101`remap` (attribe set, optional) {#tester-lycheeLinkCheck-param-remap}
102
103: An attribute set where the attribute names are regular expressions.
104 The values should be strings, derivations, or path values.
105
106 In the returned check's default configuration, external URLs are only checked when you run the `.online` attribute.
107
108 By adding remappings, you can check offline that URLs to external resources are correct, by providing a stand-in from the file system.
109
110 Before checking the existence of a URL, the regular expressions are matched and replaced by their corresponding values.
111
112 Example:
113
114 ```nix
115 {
116 "https://nix\\.dev/manual/nix/[a-z0-9.-]*" = "${nix.doc}/share/doc/nix/manual";
117 "https://nixos\\.org/manual/nix/(un)?stable" =
118 "${emptyDirectory}/placeholder-to-disallow-old-nix-docs-urls";
119 }
120 ```
121
122 Store paths in the attribute values are automatically prefixed with `file://`, because lychee requires this for paths in the file system.
123 If this is a problem, or if you need to control the order in which replacements are performed, use `extraConfig.remap` instead.
124
125`extraConfig` (attribute set) {#tester-lycheeLinkCheck-param-extraConfig}
126
127: Extra configuration to pass to `lychee` in its [configuration file](https://github.com/lycheeverse/lychee/blob/master/lychee.example.toml).
128 It is automatically [translated](https://nixos.org/manual/nixos/stable/index.html#sec-settings-nix-representable) to TOML.
129
130 Example: `{ "include_verbatim" = true; }`
131
132`lychee` (derivation, optional) {#tester-lycheeLinkCheck-param-lychee}
133
134: The `lychee` package to use.
135
136## `shellcheck` {#tester-shellcheck}
137
138Run files through `shellcheck`, a static analysis tool for shell scripts, failing if there are any issues.
139
140:::{.example #ex-shellcheck}
141# Run `testers.shellcheck`
142
143A single script
144
145```nix
146testers.shellcheck {
147 name = "script";
148 src = ./script.sh;
149}
150```
151
152Multiple files
153
154```nix
155let
156 inherit (lib) fileset;
157in
158testers.shellcheck {
159 name = "nixbsd-activate";
160 src = fileset.toSource {
161 root = ./.;
162 fileset = fileset.unions [
163 ./lib.sh
164 ./nixbsd-activate
165 ];
166 };
167}
168```
169
170:::
171
172### Inputs {#tester-shellcheck-inputs}
173
174`name` (string, optional)
175: The name of the test.
176 `name` will be required at a future point because it massively improves traceability of test failures, but is kept optional for now to avoid breaking existing usages.
177 Defaults to `run-shellcheck`.
178 The name of the derivation produced by the tester is `shellcheck-${name}` when `name` is supplied.
179
180`src` (path-like)
181: The path to the shell script(s) to check.
182 This can be a single file or a directory containing shell files.
183 All files in `src` will be checked, so you may want to provide `fileset`-based source instead of a whole directory.
184
185### Return value {#tester-shellcheck-return}
186
187A derivation that runs `shellcheck` on the given script(s), producing an empty output if no issues are found.
188The build will fail if `shellcheck` finds any issues.
189
190## `shfmt` {#tester-shfmt}
191
192Run files through `shfmt`, a shell script formatter, failing if any files are reformatted.
193
194:::{.example #ex-shfmt}
195# Run `testers.shfmt`
196
197A single script
198
199```nix
200testers.shfmt {
201 name = "script";
202 src = ./script.sh;
203}
204```
205
206Multiple files
207
208```nix
209let
210 inherit (lib) fileset;
211in
212testers.shfmt {
213 name = "nixbsd";
214 src = fileset.toSource {
215 root = ./.;
216 fileset = fileset.unions [
217 ./lib.sh
218 ./nixbsd-activate
219 ];
220 };
221}
222```
223
224:::
225
226### Inputs {#tester-shfmt-inputs}
227
228`name` (string)
229: The name of the test.
230 `name` is required because it massively improves traceability of test failures.
231 The name of the derivation produced by the tester is `shfmt-${name}`.
232
233`src` (path-like)
234: The path to the shell script(s) to check.
235 This can be a single file or a directory containing shell files.
236 All files in `src` will be checked, so you may want to provide `fileset`-based source instead of a whole directory.
237
238`indent` (integer, optional)
239: The number of spaces to use for indentation.
240 Defaults to `2`.
241 A value of `0` indents with tabs.
242
243### Return value {#tester-shfmt-return}
244
245A derivation that runs `shfmt` on the given script(s), producing an empty output upon success.
246The build will fail if `shfmt` reformats anything.
247
248## `testVersion` {#tester-testVersion}
249
250Checks that the output from running a command contains the specified version string in it as a whole word.
251
252NOTE: This is a check you add to `passthru.tests` which is mainly run by OfBorg, but not in Hydra. If you want a version check failure to block the build altogether, then [`versionCheckHook`](#versioncheckhook) is the tool you're looking for (and recommended for quick builds). The motivation for adding either of these checks would be:
253
254- Catch dynamic linking errors and such and missing environment variables that should be added by wrapping.
255- Probable protection against accidentally building the wrong version, for example when using an "old" hash in a fixed-output derivation.
256
257By default, the command to be run will be inferred from the given `package` attribute:
258it will check `meta.mainProgram` first, and fall back to `pname` or `name`.
259The default argument to the command is `--version`, and the version to be checked will be inferred from the given `package` attribute as well.
260
261:::{.example #ex-testversion-hello}
262
263# Check a program version using all the default values
264
265This example will run the command `hello --version`, and then check that the version of the `hello` package is in the output of the command.
266
267```nix
268{ passthru.tests.version = testers.testVersion { package = hello; }; }
269```
270
271:::
272
273:::{.example #ex-testversion-different-commandversion}
274
275# Check the program version using a specified command and expected version string
276
277This example will run the command `leetcode -V`, and then check that `leetcode 0.4.2` is in the output of the command as a whole word (separated by whitespaces).
278This means that an output like "leetcode 0.4.21" would fail the tests, and an output like "You're running leetcode 0.4.2" would pass the tests.
279
280A common usage of the `version` attribute is to specify `version = "v${version}"`.
281
282```nix
283{
284 version = "0.4.2";
285
286 passthru.tests.version = testers.testVersion {
287 package = leetcode-cli;
288 command = "leetcode -V";
289 version = "leetcode ${version}";
290 };
291}
292```
293
294:::
295
296## `testBuildFailure` {#tester-testBuildFailure}
297
298Make sure that a build does not succeed. This is useful for testing testers.
299
300This returns a derivation with an override on the builder, with the following effects:
301
302 - Fail the build when the original builder succeeds
303 - Move `$out` to `$out/result`, if it exists (assuming `out` is the default output)
304 - Save the build log to `$out/testBuildFailure.log` (same)
305
306While `testBuildFailure` is designed to keep changes to the original builder's environment to a minimum, some small changes are inevitable:
307
308 - The file `$TMPDIR/testBuildFailure.log` is present. It should not be deleted.
309 - `stdout` and `stderr` are a pipe instead of a tty. This could be improved.
310 - One or two extra processes are present in the sandbox during the original builder's execution.
311 - The derivation and output hashes are different, but not unusual.
312 - The derivation includes a dependency on `buildPackages.bash` and `expect-failure.sh`, which is built to include a transitive dependency on `buildPackages.coreutils` and possibly more.
313 These are not added to `PATH` or any other environment variable, so they should be hard to observe.
314
315:::{.example #ex-testBuildFailure-showingenvironmentchanges}
316
317# Check that a build fails, and verify the changes made during build
318
319```nix
320runCommand "example"
321 {
322 failed = testers.testBuildFailure (
323 runCommand "fail" { } ''
324 echo ok-ish >$out
325 echo failing though
326 exit 3
327 ''
328 );
329 }
330 ''
331 grep -F 'ok-ish' $failed/result
332 grep -F 'failing though' $failed/testBuildFailure.log
333 [[ 3 = $(cat $failed/testBuildFailure.exit) ]]
334 touch $out
335 ''
336```
337
338:::
339
340## `testBuildFailure'` {#tester-testBuildFailurePrime}
341
342This tester wraps the functionality provided by [`testers.testBuildFailure`](#tester-testBuildFailure) to make writing checks easier by simplifying checking the exit code of the builder and asserting the existence of entries in the builder's log.
343Additionally, users may specify a script containing additional checks, accessing the result of applying `testers.testBuildFailure` through the variable `failed`.
344
345NOTE: This tester will produce an empty output and exit with success if none of the checks fail; there is no need to `touch "$out"` in `script`.
346
347:::{.example #ex-testBuildFailurePrime-doc-example}
348
349# Check that a build fails, and verify the changes made during build
350
351Re-using the example from [`testers.testBuildFailure`](#ex-testBuildFailure-showingenvironmentchanges), we can see how common checks are made easier and remove the need for `runCommand`:
352
353```nix
354testers.testBuildFailure' {
355 drv = runCommand "doc-example" { } ''
356 echo ok-ish >"$out"
357 echo failing though
358 exit 3
359 '';
360 expectedBuilderExitCode = 3;
361 expectedBuilderLogEntries = [ "failing though" ];
362 script = ''
363 grep --silent -F 'ok-ish' "$failed/result"
364 '';
365}
366```
367
368:::
369
370### Inputs {#tester-testBuildFailurePrime-inputs}
371
372`drv` (derivation)
373
374: The failing derivation to wrap with `testBuildFailure`.
375
376`name` (string, optional)
377
378: The name of the test.
379 When not provided, this value defaults to `testBuildFailure-${(testers.testBuildFailure drv).name}`.
380
381`expectedBuilderExitCode` (integer, optional)
382
383: The expected exit code of the builder of `drv`.
384 When not provided, this value defaults to `1`.
385
386`expectedBuilderLogEntries` (array of string-like values, optional)
387
388: A list of string-like values which must be found in the builder's log by exact match.
389 When not provided, this value defaults to `[ ]`.
390
391 NOTE: Patterns and regular expressions are not supported.
392
393`script` (string, optional)
394
395: A string containing additional checks to run.
396 When not provided, this value defaults to `""`.
397 The result of `testers.testBuildFailure drv` is available through the variable `failed`.
398 As an example, the builder's log is at `"$failed/testBuildFailure.log"`.
399
400### Return value {#tester-testBuildFailurePrime-return}
401
402The tester produces an empty output and only succeeds when the checks using `expectedBuilderExitCode`, `expectedBuilderLogEntries`, and `script` succeed.
403
404## `testEqualContents` {#tester-testEqualContents}
405
406Check that two paths have the same contents.
407
408`assertion` (string)
409
410: A message that is printed before the comparison, after `Checking:`.
411
412`expected` (path or value coercible to store path)
413
414: The path to the expected [file system object] content
415
416`actual` (value coercible to store path) <!-- path value is possible, but wrong in practice, but let's not bother readers with our predictions -->
417
418: The path to the actual file system object content to check
419
420`postFailureMessage` (string)
421
422: A message that is printed last if the file system object contents at the two paths don't match exactly.
423
424:::{.example #ex-testEqualContents-toyexample}
425
426# Check that two paths have the same contents
427
428```nix
429testers.testEqualContents {
430 assertion = "sed -e performs replacement";
431 expected = writeText "expected" ''
432 foo baz baz
433 '';
434 actual =
435 runCommand "actual"
436 {
437 # not really necessary for a package that's in stdenv
438 nativeBuildInputs = [ gnused ];
439 base = writeText "base" ''
440 foo bar baz
441 '';
442 }
443 ''
444 sed -e 's/bar/baz/g' $base >$out
445 '';
446 # if applicable
447 postFailureMessage = ''
448 The bar-baz replacer produced an unexpected result.
449 If the new behavior is acceptable and validated against the bar-baz specification, run ./adopt-new-bar-baz-result.sh to adjust this test and require the new behavior.
450 '';
451}
452```
453
454:::
455
456## `testEqualArrayOrMap` {#tester-testEqualArrayOrMap}
457
458Check that bash arrays (including associative arrays, referred to as "maps") are populated correctly.
459
460This can be used to ensure setup hooks are registered in a certain order, or to write unit tests for shell functions which transform arrays.
461
462:::{.example #ex-testEqualArrayOrMap-test-function-add-cowbell}
463
464# Test a function which appends a value to an array
465
466```nix
467testers.testEqualArrayOrMap {
468 name = "test-function-add-cowbell";
469 valuesArray = [
470 "cowbell"
471 "cowbell"
472 ];
473 expectedArray = [
474 "cowbell"
475 "cowbell"
476 "cowbell"
477 ];
478 script = ''
479 addCowbell() {
480 local -rn arrayNameRef="$1"
481 arrayNameRef+=( "cowbell" )
482 }
483
484 nixLog "appending all values in valuesArray to actualArray"
485 for value in "''${valuesArray[@]}"; do
486 actualArray+=( "$value" )
487 done
488
489 nixLog "applying addCowbell"
490 addCowbell actualArray
491 '';
492}
493```
494
495:::
496
497### Inputs {#tester-testEqualArrayOrMap-inputs}
498
499NOTE: Internally, this tester uses `__structuredAttrs` to handle marshalling between Nix expressions and shell variables.
500This imposes the restriction that arrays and "maps" have values which are string-like.
501
502NOTE: At least one of `expectedArray` and `expectedMap` must be provided.
503
504`name` (string)
505
506: The name of the test.
507
508`script` (string)
509
510: The singular task of `script` is to populate `actualArray` or `actualMap` (it may populate both).
511 To do this, `script` may access the following shell variables:
512
513 - `valuesArray` (available when `valuesArray` is provided to the tester)
514 - `valuesMap` (available when `valuesMap` is provided to the tester)
515 - `actualArray` (available when `expectedArray` is provided to the tester)
516 - `actualMap` (available when `expectedMap` is provided to the tester)
517
518 While both `expectedArray` and `expectedMap` are in scope during the execution of `script`, they *must not* be accessed or modified from within `script`.
519
520`valuesArray` (array of string-like values, optional)
521
522: An array of string-like values.
523 This array may be used within `script`.
524
525`valuesMap` (attribute set of string-like values, optional)
526
527: An attribute set of string-like values.
528 This attribute set may be used within `script`.
529
530`expectedArray` (array of string-like values, optional)
531
532: An array of string-like values.
533 This array *must not* be accessed or modified from within `script`.
534 When provided, `script` is expected to populate `actualArray`.
535
536`expectedMap` (attribute set of string-like values, optional)
537
538: An attribute set of string-like values.
539 This attribute set *must not* be accessed or modified from within `script`.
540 When provided, `script` is expected to populate `actualMap`.
541
542### Return value {#tester-testEqualArrayOrMap-return}
543
544The tester produces an empty output and only succeeds when `expectedArray` and `expectedMap` match `actualArray` and `actualMap`, respectively, when non-null.
545The build log will contain differences encountered.
546
547## `testEqualDerivation` {#tester-testEqualDerivation}
548
549Checks that two packages produce the exact same build instructions.
550
551This can be used to make sure that a certain difference of configuration, such as the presence of an overlay does not cause a cache miss.
552
553When the derivations are equal, the return value is an empty file.
554Otherwise, the build log explains the difference via `nix-diff`.
555
556:::{.example #ex-testEqualDerivation-hello}
557
558# Check that two packages produce the same derivation
559
560```nix
561testers.testEqualDerivation "The hello package must stay the same when enabling checks." hello (
562 hello.overrideAttrs (o: {
563 doCheck = true;
564 })
565)
566```
567
568:::
569
570## `invalidateFetcherByDrvHash` {#tester-invalidateFetcherByDrvHash}
571
572Use the derivation hash to invalidate the output via name, for testing.
573
574Type: `(a@{ name, ... } -> Derivation) -> a -> Derivation`
575
576Normally, fixed output derivations can and should be cached by their output hash only, but for testing we want to re-fetch everytime the fetcher changes.
577
578Changes to the fetcher become apparent in the drvPath, which is a hash of how to fetch, rather than a fixed store path.
579By inserting this hash into the name, we can make sure to re-run the fetcher every time the fetcher changes.
580
581This relies on the assumption that Nix isn't clever enough to reuse its database of local store contents to optimize fetching.
582
583You might notice that the "salted" name derives from the normal invocation, not the final derivation.
584`invalidateFetcherByDrvHash` has to invoke the fetcher function twice:
585once to get a derivation hash, and again to produce the final fixed output derivation.
586
587:::{.example #ex-invalidateFetcherByDrvHash-nix}
588
589# Prevent nix from reusing the output of a fetcher
590
591```nix
592{
593 tests.fetchgit = testers.invalidateFetcherByDrvHash fetchgit {
594 name = "nix-source";
595 url = "https://github.com/NixOS/nix";
596 rev = "9d9dbe6ed05854e03811c361a3380e09183f4f4a";
597 hash = "sha256-7DszvbCNTjpzGRmpIVAWXk20P0/XTrWZ79KSOGLrUWY=";
598 };
599}
600```
601
602:::
603
604## `runCommand` {#tester-runCommand}
605
606`runCommand :: { name, script, stdenv ? stdenvNoCC, hash ? "...", ... } -> Derivation`
607
608This is a wrapper around `pkgs.runCommandWith`, which
609- produces a fixed-output derivation, enabling the command(s) to access the network ;
610- salts the derivation's name based on its inputs, ensuring the command is re-run whenever the inputs change.
611
612It accepts the following attributes:
613- the derivation's `name` ;
614- the `script` to be executed ;
615- `stdenv`, the environment to use, defaulting to `stdenvNoCC` ;
616- the derivation's output `hash`, defaulting to the empty file's.
617 The derivation's `outputHashMode` is set by default to recursive, so the `script` can output a directory as well.
618
619All other attributes are passed through to [`mkDerivation`](#sec-using-stdenv),
620including `nativeBuildInputs` to specify dependencies available to the `script`.
621
622:::{.example #ex-tester-runCommand-nix}
623
624# Run a command with network access
625
626```nix
627testers.runCommand {
628 name = "access-the-internet";
629 script = ''
630 curl -o /dev/null https://example.com
631 touch $out
632 '';
633 nativeBuildInputs = with pkgs; [
634 cacert
635 curl
636 ];
637}
638```
639
640:::
641
642## `runNixOSTest` {#tester-runNixOSTest}
643
644A helper function that behaves exactly like the NixOS `runTest`, except it also assigns this Nixpkgs package set as the `pkgs` of the test and makes the `nixpkgs.*` options read-only.
645
646If your test is part of the Nixpkgs repository, or if you need a more general entrypoint, see ["Calling a test" in the NixOS manual](https://nixos.org/manual/nixos/stable/index.html#sec-calling-nixos-tests).
647
648:::{.example #ex-runNixOSTest-hello}
649
650# Run a NixOS test using `runNixOSTest`
651
652```nix
653pkgs.testers.runNixOSTest (
654 { lib, ... }:
655 {
656 name = "hello";
657 nodes.machine =
658 { pkgs, ... }:
659 {
660 environment.systemPackages = [ pkgs.hello ];
661 };
662 testScript = ''
663 machine.succeed("hello")
664 '';
665 }
666)
667```
668
669:::
670
671## `nixosTest` {#tester-nixosTest}
672
673Run a NixOS VM network test using this evaluation of Nixpkgs.
674
675NOTE: This function is primarily for external use. NixOS itself uses `make-test-python.nix` directly. Packages defined in Nixpkgs [reuse NixOS tests via `nixosTests`, plural](#ssec-nixos-tests-linking).
676
677It is mostly equivalent to the function `import ./make-test-python.nix` from the [NixOS manual](https://nixos.org/nixos/manual/index.html#sec-nixos-tests), except that the current application of Nixpkgs (`pkgs`) will be used, instead of letting NixOS invoke Nixpkgs anew.
678
679If a test machine needs to set NixOS options under `nixpkgs`, it must set only the `nixpkgs.pkgs` option.
680
681### Parameter {#tester-nixosTest-parameter}
682
683A [NixOS VM test network](https://nixos.org/nixos/manual/index.html#sec-nixos-tests), or path to it. Example:
684
685```nix
686{
687 name = "my-test";
688 nodes = {
689 machine1 =
690 {
691 lib,
692 pkgs,
693 nodes,
694 ...
695 }:
696 {
697 environment.systemPackages = [ pkgs.hello ];
698 services.foo.enable = true;
699 };
700 # machine2 = ...;
701 };
702 testScript = ''
703 start_all()
704 machine1.wait_for_unit("foo.service")
705 machine1.succeed("hello | foo-send")
706 '';
707}
708```
709
710### Result {#tester-nixosTest-result}
711
712A derivation that runs the VM test.
713
714Notable attributes:
715
716 * `nodes`: the evaluated NixOS configurations. Useful for debugging and exploring the configuration.
717
718 * `driverInteractive`: a script that launches an interactive Python session in the context of the `testScript`.
719
720[file system object]: https://nix.dev/manual/nix/latest/store/file-system-object