pyrox.dev
1# Testers {#chap-testers}
2
3This chapter describes several testing builders which are available in the `testers` namespace.
4
5## `hasPkgConfigModules` {#tester-hasPkgConfigModules}
6
7<!-- Old anchor name so links still work -->
8[]{#tester-hasPkgConfigModule}
9Checks whether a package exposes a given list of `pkg-config` modules.
10If the `moduleNames` argument is omitted, `hasPkgConfigModules` will use `meta.pkgConfigModules`.
11
12:::{.example #ex-haspkgconfigmodules-defaultvalues}
13
14# Check that `pkg-config` modules are exposed using default values
15
16```nix
17{
18 passthru.tests.pkg-config = testers.hasPkgConfigModules {
19 package = finalAttrs.finalPackage;
20 };
21
22 meta.pkgConfigModules = [ "libfoo" ];
23}
24```
25
26:::
27
28:::{.example #ex-haspkgconfigmodules-explicitmodules}
29
30# Check that `pkg-config` modules are exposed using explicit module names
31
32```nix
33{
34 passthru.tests.pkg-config = testers.hasPkgConfigModules {
35 package = finalAttrs.finalPackage;
36 moduleNames = [ "libfoo" ];
37 };
38}
39```
40
41:::
42
43## `testVersion` {#tester-testVersion}
44
45Checks that the output from running a command contains the specified version string in it as a whole word.
46
47Although simplistic, this test assures that the main program can run.
48While there's no substitute for a real test case, it does catch dynamic linking errors and such.
49It also provides some protection against accidentally building the wrong version, for example when using an "old" hash in a fixed-output derivation.
50
51By default, the command to be run will be inferred from the given `package` attribute:
52it will check `meta.mainProgram` first, and fall back to `pname` or `name`.
53The default argument to the command is `--version`, and the version to be checked will be inferred from the given `package` attribute as well.
54
55:::{.example #ex-testversion-hello}
56
57# Check a program version using all the default values
58
59This example will run the command `hello --version`, and then check that the version of the `hello` package is in the output of the command.
60
61```nix
62{
63 passthru.tests.version = testers.testVersion { package = hello; };
64}
65```
66
67:::
68
69:::{.example #ex-testversion-different-commandversion}
70
71# Check the program version using a specified command and expected version string
72
73This example will run the command `leetcode -V`, and then check that `leetcode 0.4.2` is in the output of the command as a whole word (separated by whitespaces).
74This means that an output like "leetcode 0.4.21" would fail the tests, and an output like "You're running leetcode 0.4.2" would pass the tests.
75
76A common usage of the `version` attribute is to specify `version = "v${version}"`.
77
78```nix
79{
80 version = "0.4.2";
81
82 passthru.tests.version = testers.testVersion {
83 package = leetcode-cli;
84 command = "leetcode -V";
85 version = "leetcode ${version}";
86 };
87}
88```
89
90:::
91
92## `testBuildFailure` {#tester-testBuildFailure}
93
94Make sure that a build does not succeed. This is useful for testing testers.
95
96This returns a derivation with an override on the builder, with the following effects:
97
98 - Fail the build when the original builder succeeds
99 - Move `$out` to `$out/result`, if it exists (assuming `out` is the default output)
100 - Save the build log to `$out/testBuildFailure.log` (same)
101
102While `testBuildFailure` is designed to keep changes to the original builder's environment to a minimum, some small changes are inevitable:
103
104 - The file `$TMPDIR/testBuildFailure.log` is present. It should not be deleted.
105 - `stdout` and `stderr` are a pipe instead of a tty. This could be improved.
106 - One or two extra processes are present in the sandbox during the original builder's execution.
107 - The derivation and output hashes are different, but not unusual.
108 - The derivation includes a dependency on `buildPackages.bash` and `expect-failure.sh`, which is built to include a transitive dependency on `buildPackages.coreutils` and possibly more.
109 These are not added to `PATH` or any other environment variable, so they should be hard to observe.
110
111:::{.example #ex-testBuildFailure-showingenvironmentchanges}
112
113# Check that a build fails, and verify the changes made during build
114
115```nix
116runCommand "example" {
117 failed = testers.testBuildFailure (runCommand "fail" {} ''
118 echo ok-ish >$out
119 echo failing though
120 exit 3
121 '');
122} ''
123 grep -F 'ok-ish' $failed/result
124 grep -F 'failing though' $failed/testBuildFailure.log
125 [[ 3 = $(cat $failed/testBuildFailure.exit) ]]
126 touch $out
127''
128```
129
130:::
131
132## `testEqualContents` {#tester-equalContents}
133
134Check that two paths have the same contents.
135
136:::{.example #ex-testEqualContents-toyexample}
137
138# Check that two paths have the same contents
139
140```nix
141testers.testEqualContents {
142 assertion = "sed -e performs replacement";
143 expected = writeText "expected" ''
144 foo baz baz
145 '';
146 actual = runCommand "actual" {
147 # not really necessary for a package that's in stdenv
148 nativeBuildInputs = [ gnused ];
149 base = writeText "base" ''
150 foo bar baz
151 '';
152 } ''
153 sed -e 's/bar/baz/g' $base >$out
154 '';
155}
156```
157
158:::
159
160## `testEqualDerivation` {#tester-testEqualDerivation}
161
162Checks that two packages produce the exact same build instructions.
163
164This can be used to make sure that a certain difference of configuration, such as the presence of an overlay does not cause a cache miss.
165
166When the derivations are equal, the return value is an empty file.
167Otherwise, the build log explains the difference via `nix-diff`.
168
169:::{.example #ex-testEqualDerivation-hello}
170
171# Check that two packages produce the same derivation
172
173```nix
174testers.testEqualDerivation
175 "The hello package must stay the same when enabling checks."
176 hello
177 (hello.overrideAttrs(o: { doCheck = true; }))
178```
179
180:::
181
182## `invalidateFetcherByDrvHash` {#tester-invalidateFetcherByDrvHash}
183
184Use the derivation hash to invalidate the output via name, for testing.
185
186Type: `(a@{ name, ... } -> Derivation) -> a -> Derivation`
187
188Normally, fixed output derivations can and should be cached by their output hash only, but for testing we want to re-fetch everytime the fetcher changes.
189
190Changes to the fetcher become apparent in the drvPath, which is a hash of how to fetch, rather than a fixed store path.
191By inserting this hash into the name, we can make sure to re-run the fetcher every time the fetcher changes.
192
193This relies on the assumption that Nix isn't clever enough to reuse its database of local store contents to optimize fetching.
194
195You might notice that the "salted" name derives from the normal invocation, not the final derivation.
196`invalidateFetcherByDrvHash` has to invoke the fetcher function twice:
197once to get a derivation hash, and again to produce the final fixed output derivation.
198
199:::{.example #ex-invalidateFetcherByDrvHash-nix}
200
201# Prevent nix from reusing the output of a fetcher
202
203```nix
204{
205 tests.fetchgit = testers.invalidateFetcherByDrvHash fetchgit {
206 name = "nix-source";
207 url = "https://github.com/NixOS/nix";
208 rev = "9d9dbe6ed05854e03811c361a3380e09183f4f4a";
209 hash = "sha256-7DszvbCNTjpzGRmpIVAWXk20P0/XTrWZ79KSOGLrUWY=";
210 };
211}
212```
213
214:::
215
216## `runNixOSTest` {#tester-runNixOSTest}
217
218A helper function that behaves exactly like the NixOS `runTest`, except it also assigns this Nixpkgs package set as the `pkgs` of the test and makes the `nixpkgs.*` options read-only.
219
220If your test is part of the Nixpkgs repository, or if you need a more general entrypoint, see ["Calling a test" in the NixOS manual](https://nixos.org/manual/nixos/stable/index.html#sec-calling-nixos-tests).
221
222:::{.example #ex-runNixOSTest-hello}
223
224# Run a NixOS test using `runNixOSTest`
225
226```nix
227pkgs.testers.runNixOSTest ({ lib, ... }: {
228 name = "hello";
229 nodes.machine = { pkgs, ... }: {
230 environment.systemPackages = [ pkgs.hello ];
231 };
232 testScript = ''
233 machine.succeed("hello")
234 '';
235})
236```
237
238:::
239
240## `nixosTest` {#tester-nixosTest}
241
242Run a NixOS VM network test using this evaluation of Nixpkgs.
243
244NOTE: This function is primarily for external use. NixOS itself uses `make-test-python.nix` directly. Packages defined in Nixpkgs [reuse NixOS tests via `nixosTests`, plural](#ssec-nixos-tests-linking).
245
246It is mostly equivalent to the function `import ./make-test-python.nix` from the [NixOS manual](https://nixos.org/nixos/manual/index.html#sec-nixos-tests), except that the current application of Nixpkgs (`pkgs`) will be used, instead of letting NixOS invoke Nixpkgs anew.
247
248If a test machine needs to set NixOS options under `nixpkgs`, it must set only the `nixpkgs.pkgs` option.
249
250### Parameter {#tester-nixosTest-parameter}
251
252A [NixOS VM test network](https://nixos.org/nixos/manual/index.html#sec-nixos-tests), or path to it. Example:
253
254```nix
255{
256 name = "my-test";
257 nodes = {
258 machine1 = { lib, pkgs, nodes, ... }: {
259 environment.systemPackages = [ pkgs.hello ];
260 services.foo.enable = true;
261 };
262 # machine2 = ...;
263 };
264 testScript = ''
265 start_all()
266 machine1.wait_for_unit("foo.service")
267 machine1.succeed("hello | foo-send")
268 '';
269}
270```
271
272### Result {#tester-nixosTest-result}
273
274A derivation that runs the VM test.
275
276Notable attributes:
277
278 * `nodes`: the evaluated NixOS configurations. Useful for debugging and exploring the configuration.
279
280 * `driverInteractive`: a script that launches an interactive Python session in the context of the `testScript`.