pyrox.dev
1# Testers {#chap-testers}
2This chapter describes several testing builders which are available in the `testers` namespace.
3
4## `hasPkgConfigModules` {#tester-hasPkgConfigModules}
5
6<!-- Old anchor name so links still work -->
7[]{#tester-hasPkgConfigModule}
8Checks whether a package exposes a given list of `pkg-config` modules.
9If the `moduleNames` argument is omitted, `hasPkgConfigModules` will
10use `meta.pkgConfigModules`.
11
12Example:
13
14```nix
15passthru.tests.pkg-config = testers.hasPkgConfigModules {
16 package = finalAttrs.finalPackage;
17 moduleNames = [ "libfoo" ];
18};
19```
20
21If the package in question has `meta.pkgConfigModules` set, it is even simpler:
22
23```nix
24passthru.tests.pkg-config = testers.hasPkgConfigModules {
25 package = finalAttrs.finalPackage;
26};
27
28meta.pkgConfigModules = [ "libfoo" ];
29```
30
31## `testVersion` {#tester-testVersion}
32
33Checks the command output contains the specified version
34
35Although simplistic, this test assures that the main program
36can run. While there's no substitute for a real test case,
37it does catch dynamic linking errors and such. It also provides
38some protection against accidentally building the wrong version,
39for example when using an 'old' hash in a fixed-output derivation.
40
41Examples:
42
43```nix
44passthru.tests.version = testers.testVersion { package = hello; };
45
46passthru.tests.version = testers.testVersion {
47 package = seaweedfs;
48 command = "weed version";
49};
50
51passthru.tests.version = testers.testVersion {
52 package = key;
53 command = "KeY --help";
54 # Wrong '2.5' version in the code. Drop on next version.
55 version = "2.5";
56};
57
58passthru.tests.version = testers.testVersion {
59 package = ghr;
60 # The output needs to contain the 'version' string without any prefix or suffix.
61 version = "v${version}";
62};
63```
64
65## `testBuildFailure` {#tester-testBuildFailure}
66
67Make sure that a build does not succeed. This is useful for testing testers.
68
69This returns a derivation with an override on the builder, with the following effects:
70
71 - Fail the build when the original builder succeeds
72 - Move `$out` to `$out/result`, if it exists (assuming `out` is the default output)
73 - Save the build log to `$out/testBuildFailure.log` (same)
74
75Example:
76
77```nix
78runCommand "example" {
79 failed = testers.testBuildFailure (runCommand "fail" {} ''
80 echo ok-ish >$out
81 echo failing though
82 exit 3
83 '');
84} ''
85 grep -F 'ok-ish' $failed/result
86 grep -F 'failing though' $failed/testBuildFailure.log
87 [[ 3 = $(cat $failed/testBuildFailure.exit) ]]
88 touch $out
89'';
90```
91
92While `testBuildFailure` is designed to keep changes to the original builder's
93environment to a minimum, some small changes are inevitable.
94
95 - The file `$TMPDIR/testBuildFailure.log` is present. It should not be deleted.
96 - `stdout` and `stderr` are a pipe instead of a tty. This could be improved.
97 - One or two extra processes are present in the sandbox during the original
98 builder's execution.
99 - The derivation and output hashes are different, but not unusual.
100 - The derivation includes a dependency on `buildPackages.bash` and
101 `expect-failure.sh`, which is built to include a transitive dependency on
102 `buildPackages.coreutils` and possibly more. These are not added to `PATH`
103 or any other environment variable, so they should be hard to observe.
104
105## `testEqualContents` {#tester-equalContents}
106
107Check that two paths have the same contents.
108
109Example:
110
111```nix
112testers.testEqualContents {
113 assertion = "sed -e performs replacement";
114 expected = writeText "expected" ''
115 foo baz baz
116 '';
117 actual = runCommand "actual" {
118 # not really necessary for a package that's in stdenv
119 nativeBuildInputs = [ gnused ];
120 base = writeText "base" ''
121 foo bar baz
122 '';
123 } ''
124 sed -e 's/bar/baz/g' $base >$out
125 '';
126}
127```
128
129## `testEqualDerivation` {#tester-testEqualDerivation}
130
131Checks that two packages produce the exact same build instructions.
132
133This can be used to make sure that a certain difference of configuration,
134such as the presence of an overlay does not cause a cache miss.
135
136When the derivations are equal, the return value is an empty file.
137Otherwise, the build log explains the difference via `nix-diff`.
138
139Example:
140
141```nix
142testers.testEqualDerivation
143 "The hello package must stay the same when enabling checks."
144 hello
145 (hello.overrideAttrs(o: { doCheck = true; }))
146```
147
148## `invalidateFetcherByDrvHash` {#tester-invalidateFetcherByDrvHash}
149
150Use the derivation hash to invalidate the output via name, for testing.
151
152Type: `(a@{ name, ... } -> Derivation) -> a -> Derivation`
153
154Normally, fixed output derivations can and should be cached by their output
155hash only, but for testing we want to re-fetch everytime the fetcher changes.
156
157Changes to the fetcher become apparent in the drvPath, which is a hash of
158how to fetch, rather than a fixed store path.
159By inserting this hash into the name, we can make sure to re-run the fetcher
160every time the fetcher changes.
161
162This relies on the assumption that Nix isn't clever enough to reuse its
163database of local store contents to optimize fetching.
164
165You might notice that the "salted" name derives from the normal invocation,
166not the final derivation. `invalidateFetcherByDrvHash` has to invoke the fetcher
167function twice: once to get a derivation hash, and again to produce the final
168fixed output derivation.
169
170Example:
171
172```nix
173tests.fetchgit = testers.invalidateFetcherByDrvHash fetchgit {
174 name = "nix-source";
175 url = "https://github.com/NixOS/nix";
176 rev = "9d9dbe6ed05854e03811c361a3380e09183f4f4a";
177 hash = "sha256-7DszvbCNTjpzGRmpIVAWXk20P0/XTrWZ79KSOGLrUWY=";
178};
179```
180
181## `runNixOSTest` {#tester-runNixOSTest}
182
183A helper function that behaves exactly like the NixOS `runTest`, except it also assigns this Nixpkgs package set as the `pkgs` of the test and makes the `nixpkgs.*` options read-only.
184
185If your test is part of the Nixpkgs repository, or if you need a more general entrypoint, see ["Calling a test" in the NixOS manual](https://nixos.org/manual/nixos/stable/index.html#sec-calling-nixos-tests).
186
187Example:
188
189```nix
190pkgs.testers.runNixOSTest ({ lib, ... }: {
191 name = "hello";
192 nodes.machine = { pkgs, ... }: {
193 environment.systemPackages = [ pkgs.hello ];
194 };
195 testScript = ''
196 machine.succeed("hello")
197 '';
198})
199```
200
201## `nixosTest` {#tester-nixosTest}
202
203Run a NixOS VM network test using this evaluation of Nixpkgs.
204
205NOTE: This function is primarily for external use. NixOS itself uses `make-test-python.nix` directly. Packages defined in Nixpkgs [reuse NixOS tests via `nixosTests`, plural](#ssec-nixos-tests-linking).
206
207It is mostly equivalent to the function `import ./make-test-python.nix` from the
208[NixOS manual](https://nixos.org/nixos/manual/index.html#sec-nixos-tests),
209except that the current application of Nixpkgs (`pkgs`) will be used, instead of
210letting NixOS invoke Nixpkgs anew.
211
212If a test machine needs to set NixOS options under `nixpkgs`, it must set only the
213`nixpkgs.pkgs` option.
214
215### Parameter {#tester-nixosTest-parameter}
216
217A [NixOS VM test network](https://nixos.org/nixos/manual/index.html#sec-nixos-tests), or path to it. Example:
218
219```nix
220{
221 name = "my-test";
222 nodes = {
223 machine1 = { lib, pkgs, nodes, ... }: {
224 environment.systemPackages = [ pkgs.hello ];
225 services.foo.enable = true;
226 };
227 # machine2 = ...;
228 };
229 testScript = ''
230 start_all()
231 machine1.wait_for_unit("foo.service")
232 machine1.succeed("hello | foo-send")
233 '';
234}
235```
236
237### Result {#tester-nixosTest-result}
238
239A derivation that runs the VM test.
240
241Notable attributes:
242
243 * `nodes`: the evaluated NixOS configurations. Useful for debugging and exploring the configuration.
244
245 * `driverInteractive`: a script that launches an interactive Python session in the context of the `testScript`.