pyrox.dev
1# Fetchers {#chap-pkgs-fetchers}
2
3Building software with Nix often requires downloading source code and other files from the internet.
4To this end, we use functions that we call _fetchers_, which obtain remote sources via various protocols and services.
5
6Nix provides built-in fetchers such as [`fetchTarball`](https://nixos.org/manual/nix/stable/language/builtins.html#builtins-fetchTarball).
7Nixpkgs provides its own fetchers, which work differently:
8
9- A built-in fetcher will download and cache files at evaluation time and produce a [store path](https://nixos.org/manual/nix/stable/glossary#gloss-store-path).
10 A Nixpkgs fetcher will create a ([fixed-output](https://nixos.org/manual/nix/stable/glossary#gloss-fixed-output-derivation)) [derivation](https://nixos.org/manual/nix/stable/glossary#gloss-derivation), and files are downloaded at build time.
11- Built-in fetchers will invalidate their cache after [`tarball-ttl`](https://nixos.org/manual/nix/stable/command-ref/conf-file#conf-tarball-ttl) expires, and will require network activity to check if the cache entry is up to date.
12 Nixpkgs fetchers only re-download if the specified hash changes or the store object is not available.
13- Built-in fetchers do not use [substituters](https://nixos.org/manual/nix/stable/command-ref/conf-file#conf-substituters).
14 Derivations produced by Nixpkgs fetchers will use any configured binary cache transparently.
15
16This significantly reduces the time needed to evaluate Nixpkgs, and allows [Hydra](https://nixos.org/hydra) to retain and re-distribute sources used by Nixpkgs in the [public binary cache](https://cache.nixos.org).
17For these reasons, Nix's built-in fetchers are not allowed in Nixpkgs.
18
19The following table summarises the differences:
20
21| Fetchers | Download | Output | Cache | Re-download when |
22|-|-|-|-|-|
23| `builtins.fetch*` | evaluation time | store path | `/nix/store`, `~/.cache/nix` | `tarball-ttl` expires, cache miss in `~/.cache/nix`, output store object not in local store |
24| `pkgs.fetch*` | build time | derivation | `/nix/store`, substituters | output store object not available |
25
26:::{.tip}
27`pkgs.fetchFrom*` helpers retrieve _snapshots_ of version-controlled sources, as opposed to the entire version history, which is more efficient.
28`pkgs.fetchgit` by default also has the same behaviour, but can be changed through specific attributes given to it.
29:::
30
31## Caveats {#chap-pkgs-fetchers-caveats}
32
33Because Nixpkgs fetchers are fixed-output derivations, an [output hash](https://nixos.org/manual/nix/stable/language/advanced-attributes#adv-attr-outputHash) has to be specified, usually indirectly through a `hash` attribute.
34This hash refers to the derivation output, which can be different from the remote source itself!
35
36This has the following implications that you should be aware of:
37
38- Use Nix (or Nix-aware) tooling to produce the output hash.
39
40- When changing any fetcher parameters, always update the output hash.
41 Use one of the methods from [](#sec-pkgs-fetchers-updating-source-hashes).
42 Otherwise, existing store objects that match the output hash will be re-used rather than fetching new content.
43
44 :::{.note}
45 A similar problem arises while testing changes to a fetcher's implementation.
46 If the output of the derivation already exists in the Nix store, test failures can go undetected.
47 The [`invalidateFetcherByDrvHash`](#tester-invalidateFetcherByDrvHash) function helps prevent reusing cached derivations.
48 :::
49
50## Updating source hashes {#sec-pkgs-fetchers-updating-source-hashes}
51
52There are several ways to obtain the hash corresponding to a remote source.
53Unless you understand how the fetcher you're using calculates the hash from the downloaded contents, you should use [the fake hash method](#sec-pkgs-fetchers-updating-source-hashes-fakehash-method).
54
551. []{#sec-pkgs-fetchers-updating-source-hashes-fakehash-method} The fake hash method: In your package recipe, set the hash to one of
56
57 - `""`
58 - `lib.fakeHash`
59 - `lib.fakeSha256`
60 - `lib.fakeSha512`
61
62 Attempt to build, extract the calculated hashes from error messages, and put them into the recipe.
63
64 :::{.warning}
65 You must use one of these four fake hashes and not some arbitrarily-chosen hash.
66 See [](#sec-pkgs-fetchers-secure-hashes) for details.
67 :::
68
69 :::{.example #ex-fetchers-update-fod-hash}
70 # Update source hash with the fake hash method
71
72 Consider the following recipe that produces a plain file:
73
74 ```nix
75 { fetchurl }:
76 fetchurl {
77 url = "https://raw.githubusercontent.com/NixOS/nixpkgs/23.05/.version";
78 hash = "sha256-ZHl1emidXVojm83LCVrwULpwIzKE/mYwfztVkvpruOM=";
79 }
80 ```
81
82 A common mistake is to update a fetcher parameter, such as `url`, without updating the hash:
83
84 ```nix
85 { fetchurl }:
86 fetchurl {
87 url = "https://raw.githubusercontent.com/NixOS/nixpkgs/23.11/.version";
88 hash = "sha256-ZHl1emidXVojm83LCVrwULpwIzKE/mYwfztVkvpruOM=";
89 }
90 ```
91
92 **This will produce the same output as before!**
93 Set the hash to an empty string:
94
95 ```nix
96 { fetchurl }:
97 fetchurl {
98 url = "https://raw.githubusercontent.com/NixOS/nixpkgs/23.11/.version";
99 hash = "";
100 }
101 ```
102
103 When building the package, use the error message to determine the correct hash:
104
105 ```shell
106 $ nix-build
107 (some output removed for clarity)
108 error: hash mismatch in fixed-output derivation '/nix/store/7yynn53jpc93l76z9zdjj4xdxgynawcw-version.drv':
109 specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
110 got: sha256-BZqI7r0MNP29yGH5+yW2tjU9OOpOCEvwWKrWCv5CQ0I=
111 error: build of '/nix/store/bqdjcw5ij5ymfbm41dq230chk9hdhqff-version.drv' failed
112 ```
113 :::
114
1152. Prefetch the source with [`nix-prefetch-<type> <URL>`](https://search.nixos.org/packages?buckets={%22package_attr_set%22%3A[%22No%20package%20set%22]%2C%22package_license_set%22%3A[]%2C%22package_maintainers_set%22%3A[]%2C%22package_platforms%22%3A[]}&query=nix-prefetch), where `<type>` is one of
116
117 - `url`
118 - `git`
119 - `hg`
120 - `cvs`
121 - `bzr`
122 - `svn`
123
124 The hash is printed to stdout.
125
1263. Prefetch by package source (with `nix-prefetch-url '<nixpkgs>' -A <package>.src`, where `<package>` is package attribute name).
127 The hash is printed to stdout.
128
129 This works well when you've upgraded the existing package version and want to find out new hash, but is useless if the package can't be accessed by attribute or the package has multiple sources (`.srcs`, architecture-dependent sources, etc).
130
1314. Upstream hash: use it when upstream provides `sha256` or `sha512`.
132 Don't use it when upstream provides `md5`, compute `sha256` instead.
133
134 A little nuance is that `nix-prefetch-*` tools produce hashes with the `nix32` encoding (a Nix-specific base32 adaptation), but upstream usually provides hexadecimal (`base16`) encoding.
135 Fetchers understand both formats.
136 Nixpkgs does not standardise on any one format.
137
138 You can convert between hash formats with [`nix-hash`](https://nixos.org/manual/nix/stable/command-ref/nix-hash).
139
1405. Extract the hash from a local source archive with `sha256sum`.
141 Use `nix-prefetch-url file:///path/to/archive` if you want the custom Nix `base32` hash.
142
143## Obtaining hashes securely {#sec-pkgs-fetchers-secure-hashes}
144
145It's always a good idea to avoid Man-in-the-Middle (MITM) attacks when downloading source contents.
146Otherwise, you could unknowingly download malware instead of the intended source, and instead of the actual source hash, you'll end up using the hash of malware.
147Here are security considerations for this scenario:
148
149- `http://` URLs are not secure to prefetch hashes.
150
151- Upstream hashes should be obtained via a secure protocol.
152
153- `https://` URLs give you more protections when using `nix-prefetch-*` or for upstream hashes.
154
155- `https://` URLs are secure when using the [fake hash method](#sec-pkgs-fetchers-updating-source-hashes-fakehash-method) *only if* you use one of the listed fake hashes.
156 If you use any other hash, the download will be exposed to MITM attacks even if you use HTTPS URLs.
157
158 In more concrete terms, if you use any other hash, the [`--insecure` flag](https://curl.se/docs/manpage.html#-k) will be passed to the underlying call to `curl` when downloading content.
159
160## Proxy usage {#sec-pkgs-fetchers-proxy}
161
162Nixpkgs fetchers can make use of a http(s) proxy. Each fetcher will automatically inherit proxy-related environment variables (`http_proxy`, `https_proxy`, etc) via [impureEnvVars](https://nixos.org/manual/nix/stable/language/advanced-attributes#adv-attr-impureEnvVars).
163
164The environment variable `NIX_SSL_CERT_FILE` is also inherited in fetchers, and can be used to provide a custom certificate bundle to fetchers. This is usually required for a https proxy to work without certificate validation errors.
165
166To use a temporary Tor instance as a proxy for fetching from `.onion` addresses, add `nativeBuildInputs = [ tor.proxyHook ];` to the fetcher parameters.
167
168[]{#fetchurl}
169## `fetchurl` {#sec-pkgs-fetchers-fetchurl}
170
171`fetchurl` returns a [fixed-output derivation](https://nixos.org/manual/nix/stable/glossary.html#gloss-fixed-output-derivation) which downloads content from a given URL and stores the unaltered contents within the Nix store.
172
173It uses {manpage}`curl(1)` internally, and allows its behaviour to be modified by specifying a few attributes in the argument to `fetchurl` (see the documentation for attributes `curlOpts`, `curlOptsList`, and `netrcPhase`).
174
175The resulting [store path](https://nixos.org/manual/nix/stable/store/store-path) is determined by the hash given to `fetchurl`, and also the `name` (or `pname` and `version`) values.
176
177If neither `name` nor `pname` and `version` are specified when calling `fetchurl`, it will default to using the [basename](https://nixos.org/manual/nix/stable/language/builtins.html#builtins-baseNameOf) of `url` or the first element of `urls`.
178If `pname` and `version` are specified, `fetchurl` will use those values and will ignore `name`, even if it is also specified.
179
180### Inputs {#sec-pkgs-fetchers-fetchurl-inputs}
181
182`fetchurl` requires an attribute set with the following attributes:
183
184`url` (String; _optional_)
185: The URL to download from.
186
187 :::{.note}
188 Either `url` or `urls` must be specified, but not both.
189 :::
190
191 All URLs of the format [specified here](https://curl.se/docs/url-syntax.html#rfc-3986-plus) are supported.
192
193 _Default value:_ `""`.
194
195`urls` (List of String; _optional_)
196: A list of URLs, specifying download locations for the same content.
197 Each URL will be tried in order until one of them succeeds with some content or all of them fail.
198 See [](#ex-fetchers-fetchurl-nixpkgs-version-multiple-urls) to understand how this attribute affects the behaviour of `fetchurl`.
199
200 :::{.note}
201 Either `url` or `urls` must be specified, but not both.
202 :::
203
204 _Default value:_ `[]`.
205
206`hash` (String; _optional_)
207: Hash of the derivation output of `fetchurl`, following the format for integrity metadata as defined by [SRI](https://www.w3.org/TR/SRI/).
208 For more information, see [](#chap-pkgs-fetchers-caveats).
209
210 :::{.note}
211 It is recommended that you use the `hash` attribute instead of the other hash-specific attributes that exist for backwards compatibility.
212
213 If `hash` is not specified, you must specify `outputHash` and `outputHashAlgo`, or one of `sha512`, `sha256`, or `sha1`.
214 :::
215
216 _Default value:_ `""`.
217
218`outputHash` (String; _optional_)
219: Hash of the derivation output of `fetchurl` in the format expected by Nix.
220 See [the documentation on the Nix manual](https://nixos.org/manual/nix/stable/language/advanced-attributes.html#adv-attr-outputHash) for more information about its format.
221
222 :::{.note}
223 It is recommended that you use the `hash` attribute instead.
224
225 If `outputHash` is specified, you must also specify `outputHashAlgo`.
226 :::
227
228 _Default value:_ `""`.
229
230`outputHashAlgo` (String; _optional_)
231: Algorithm used to generate the value specified in `outputHash`.
232 See [the documentation on the Nix manual](https://nixos.org/manual/nix/stable/language/advanced-attributes.html#adv-attr-outputHashAlgo) for more information about the values it supports.
233
234 :::{.note}
235 It is recommended that you use the `hash` attribute instead.
236
237 The value specified in `outputHashAlgo` will be ignored if `outputHash` isn't also specified.
238 :::
239
240 _Default value:_ `""`.
241
242`sha1` (String; _optional_)
243: SHA-1 hash of the derivation output of `fetchurl` in the format expected by Nix.
244 See [the documentation on the Nix manual](https://nixos.org/manual/nix/stable/language/advanced-attributes.html#adv-attr-outputHash) for more information about its format.
245
246 :::{.note}
247 It is recommended that you use the `hash` attribute instead.
248 :::
249
250 _Default value:_ `""`.
251
252`sha256` (String; _optional_)
253: SHA-256 hash of the derivation output of `fetchurl` in the format expected by Nix.
254 See [the documentation on the Nix manual](https://nixos.org/manual/nix/stable/language/advanced-attributes.html#adv-attr-outputHash) for more information about its format.
255
256 :::{.note}
257 It is recommended that you use the `hash` attribute instead.
258 :::
259
260 _Default value:_ `""`.
261
262`sha512` (String; _optional_)
263: SHA-512 hash of the derivation output of `fetchurl` in the format expected by Nix.
264 See [the documentation on the Nix manual](https://nixos.org/manual/nix/stable/language/advanced-attributes.html#adv-attr-outputHash) for more information about its format.
265
266 :::{.note}
267 It is recommended that you use the `hash` attribute instead.
268 :::
269
270 _Default value:_ `""`.
271
272`name` (String; _optional_)
273: The symbolic name of the downloaded file when saved in the Nix store.
274 See [the `fetchurl` overview](#sec-pkgs-fetchers-fetchurl) for details on how the name of the file is decided.
275
276 _Default value:_ `""`.
277
278`pname` (String; _optional_)
279: A base name, which will be combined with `version` to form the symbolic name of the downloaded file when saved in the Nix store.
280 See [the `fetchurl` overview](#sec-pkgs-fetchers-fetchurl) for details on how the name of the file is decided.
281
282 :::{.note}
283 If `pname` is specified, you must also specify `version`, otherwise `fetchurl` will ignore the value of `pname`.
284 :::
285
286 _Default value:_ `""`.
287
288`version` (String; _optional_)
289: A version, which will be combined with `pname` to form the symbolic name of the downloaded file when saved in the Nix store.
290 See [the `fetchurl` overview](#sec-pkgs-fetchers-fetchurl) for details on how the name of the file is decided.
291
292 _Default value:_ `""`.
293
294`recursiveHash` (Boolean; _optional_) []{#sec-pkgs-fetchers-fetchurl-inputs-recursiveHash}
295: If set to `true`, will signal to Nix that the hash given to `fetchurl` was calculated using the `"recursive"` mode.
296 See [the documentation on the Nix manual](https://nixos.org/manual/nix/stable/language/advanced-attributes.html#adv-attr-outputHashMode) for more information about the existing modes.
297
298 By default, `fetchurl` uses `"recursive"` mode when the `executable` attribute is set to `true`, so you don't need to specify `recursiveHash` in this case.
299
300 _Default value:_ `false`.
301
302`executable` (Boolean; _optional_)
303: If `true`, sets the executable bit on the downloaded file.
304
305 _Default value_: `false`.
306
307`downloadToTemp` (Boolean; _optional_) []{#sec-pkgs-fetchers-fetchurl-inputs-downloadToTemp}
308: If `true`, saves the downloaded file to a temporary location instead of the expected Nix store location.
309 This is useful when used in conjunction with `postFetch` attribute, otherwise `fetchurl` will not produce any meaningful output.
310
311 The location of the downloaded file will be set in the `$downloadedFile` variable, which should be used by the script in the `postFetch` attribute.
312 See [](#ex-fetchers-fetchurl-nixpkgs-version-postfetch) to understand how to work with this attribute.
313
314 _Default value:_ `false`.
315
316`postFetch` (String; _optional_)
317: Script executed after the file has been downloaded successfully, and before `fetchurl` finishes running.
318 Useful for post-processing, to check or transform the file in some way.
319 See [](#ex-fetchers-fetchurl-nixpkgs-version-postfetch) to understand how to work with this attribute.
320
321 _Default value:_ `""`.
322
323`netrcPhase` (String or Null; _optional_)
324: Script executed to create a {manpage}`netrc(5)` file to be used with {manpage}`curl(1)`.
325 The script should create the `netrc` file (note that it does not begin with a ".") in the directory it's currently running in (`$PWD`).
326
327 The script is executed during the setup done by `fetchurl` before it runs any of its code to download the specified content.
328
329 :::{.note}
330 If specified, `fetchurl` will automatically alter its invocation of {manpage}`curl(1)` to use the `netrc` file, so you don't need to add anything to `curlOpts` or `curlOptsList`.
331 :::
332
333 :::{.caution}
334 Since `netrcPhase` needs to be specified in your source Nix code, any secrets that you put directly in it will be world-readable by design (both in your source code, and when the derivation gets created in the Nix store).
335
336 If you want to avoid this behaviour, see the documentation of `netrcImpureEnvVars` for an alternative way of dealing with these secrets.
337 :::
338
339 _Default value_: `null`.
340
341`netrcImpureEnvVars` (List of String; _optional_)
342: If specified, `fetchurl` will add these environment variable names to the list of [impure environment variables](https://nixos.org/manual/nix/stable/language/advanced-attributes.html#adv-attr-impureEnvVars), which will be passed from the environment of the calling user to the builder running the `fetchurl` code.
343
344 This is useful when used with `netrcPhase` to hide any secrets that are used in it, because the script in `netrcPhase` only needs to reference the environment variables with the secrets in them instead.
345 However, note that these are called _impure_ variables for a reason:
346 the environment that starts the build needs to have these variables declared for everything to work properly, which means that additional setup is required outside what Nix controls.
347
348 _Default value:_ `[]`.
349
350`curlOpts` (String; _optional_)
351: If specified, this value will be appended to the invocation of {manpage}`curl(1)` when downloading the URL(s) given to `fetchurl`.
352 Multiple arguments can be separated by spaces normally, but values with whitespaces will be interpreted as multiple arguments (instead of a single value), even if the value is escaped.
353 See `curlOptsList` for a way to pass values with whitespaces in them.
354
355 _Default value:_ `""`.
356
357`curlOptsList` (List of String; _optional_)
358: If specified, each element of this list will be passed as an argument to the invocation of {manpage}`curl(1)` when downloading the URL(s) given to `fetchurl`.
359 This allows passing values that contain spaces, with no escaping needed.
360
361 _Default value:_ `[]`.
362
363`showURLs` (Boolean; _optional_)
364: If set to `true`, this will stop `fetchurl` from downloading anything at all.
365 Instead, it will output a list of all the URLs it would've used to download the content (after resolving `mirror://` URLs, for example).
366 This is useful for debugging.
367
368 _Default value:_ `false`.
369
370`meta` (Attribute Set; _optional_)
371: Specifies any [meta-attributes](#chap-meta) for the derivation returned by `fetchurl`.
372
373 _Default value:_ `{}`.
374
375`passthru` (Attribute Set; _optional_)
376: Specifies any extra [`passthru`](#chap-passthru) attributes for the derivation returned by `fetchurl`.
377 Note that `fetchurl` defines [`passthru` attributes of its own](#ssec-pkgs-fetchers-fetchurl-passthru-outputs).
378 Attributes specified in `passthru` can override the default attributes returned by `fetchurl`.
379
380 _Default value:_ `{}`.
381
382`preferLocalBuild` (Boolean; _optional_)
383: This is the same attribute as [defined in the Nix manual](https://nixos.org/manual/nix/stable/language/advanced-attributes.html#adv-attr-preferLocalBuild).
384 It is `true` by default because making a remote machine download the content just duplicates network traffic (since the local machine might download the results from the derivation anyway), but this could be useful in cases where network access is restricted on local machines.
385
386 _Default value:_ `true`.
387
388`nativeBuildInputs` (List of Attribute Set; _optional_)
389: Additional packages needed to download the content.
390 This is useful if you need extra packages for `postFetch` or `netrcPhase`, for example.
391 Has the same semantics as in [](#var-stdenv-nativeBuildInputs).
392 See [](#ex-fetchers-fetchurl-nixpkgs-version-postfetch) to understand how this can be used with `postFetch`.
393
394 _Default value:_ `[]`.
395
396### Passthru outputs {#ssec-pkgs-fetchers-fetchurl-passthru-outputs}
397
398`fetchurl` also defines its own [`passthru`](#chap-passthru) attributes:
399
400`url` (String)
401
402: The same `url` attribute passed in the argument to `fetchurl`.
403
404### Examples {#ssec-pkgs-fetchers-fetchurl-examples}
405
406:::{.example #ex-fetchers-fetchurl-nixpkgs-version}
407# Using `fetchurl` to download a file
408
409The following package downloads a small file from a URL and shows the most common way to use `fetchurl`:
410
411```nix
412{ fetchurl }:
413fetchurl {
414 url = "https://raw.githubusercontent.com/NixOS/nixpkgs/23.11/.version";
415 hash = "sha256-BZqI7r0MNP29yGH5+yW2tjU9OOpOCEvwWKrWCv5CQ0I=";
416}
417```
418
419After building the package, the file will be downloaded and place into the Nix store:
420
421```shell
422$ nix-build
423(output removed for clarity)
424/nix/store/4g9y3x851wqrvim4zcz5x2v3zivmsq8n-version
425
426$ cat /nix/store/4g9y3x851wqrvim4zcz5x2v3zivmsq8n-version
42723.11
428```
429:::
430
431:::{.example #ex-fetchers-fetchurl-nixpkgs-version-multiple-urls}
432# Using `fetchurl` to download a file with multiple possible URLs
433
434The following package adapts [](#ex-fetchers-fetchurl-nixpkgs-version) to use multiple URLs.
435The first URL was crafted to intentionally return an error to illustrate how `fetchurl` will try multiple URLs until it finds one that works (or all URLs fail).
436
437```nix
438{ fetchurl }:
439fetchurl {
440 urls = [
441 "https://raw.githubusercontent.com/NixOS/nixpkgs/23.11/does-not-exist"
442 "https://raw.githubusercontent.com/NixOS/nixpkgs/23.11/.version"
443 ];
444 hash = "sha256-BZqI7r0MNP29yGH5+yW2tjU9OOpOCEvwWKrWCv5CQ0I=";
445}
446```
447
448After building the package, both URLs will be used to download the file:
449
450```shell
451$ nix-build
452(some output removed for clarity)
453trying https://raw.githubusercontent.com/NixOS/nixpkgs/23.11/does-not-exist
454(some output removed for clarity)
455curl: (22) The requested URL returned error: 404
456
457trying https://raw.githubusercontent.com/NixOS/nixpkgs/23.11/.version
458(some output removed for clarity)
459/nix/store/n9asny31z32q7sdw6a8r1gllrsfy53kl-does-not-exist
460
461$ cat /nix/store/n9asny31z32q7sdw6a8r1gllrsfy53kl-does-not-exist
46223.11
463```
464
465However, note that the name of the file was derived from the first URL (this is further explained in [the `fetchurl` overview](#sec-pkgs-fetchers-fetchurl)).
466To ensure the result will have the same name regardless of which URLs are used, we can modify the package:
467
468```nix
469{ fetchurl }:
470fetchurl {
471 name = "nixpkgs-version";
472 urls = [
473 "https://raw.githubusercontent.com/NixOS/nixpkgs/23.11/does-not-exist"
474 "https://raw.githubusercontent.com/NixOS/nixpkgs/23.11/.version"
475 ];
476 hash = "sha256-BZqI7r0MNP29yGH5+yW2tjU9OOpOCEvwWKrWCv5CQ0I=";
477}
478```
479
480After building the package, the result will have the name we specified:
481
482```shell
483$ nix-build
484(output removed for clarity)
485/nix/store/zczb6wl3al6jm9sm5h3pr6nqn0i5ji9z-nixpkgs-version
486```
487:::
488
489:::{.example #ex-fetchers-fetchurl-nixpkgs-version-postfetch}
490# Manipulating the content downloaded by `fetchurl`
491
492It might be useful to manipulate the content downloaded by `fetchurl` directly in its derivation.
493In this example, we'll adapt [](#ex-fetchers-fetchurl-nixpkgs-version) to append the result of running the `hello` package to the contents we download, purely to illustrate how to manipulate the content.
494
495```nix
496{
497 fetchurl,
498 hello,
499 lib,
500}:
501fetchurl {
502 url = "https://raw.githubusercontent.com/NixOS/nixpkgs/23.11/.version";
503
504 nativeBuildInputs = [ hello ];
505
506 downloadToTemp = true;
507 postFetch = ''
508 ${lib.getExe hello} >> $downloadedFile
509 mv $downloadedFile $out
510 '';
511
512 hash = "sha256-ceooQQYmDx5+0nfg40uU3NNI2yKrixP7HZ/xLZUNv+w=";
513}
514```
515
516After building the package, the resulting file will have "Hello, world!" appended to it:
517
518```shell
519$ nix-build
520(output removed for clarity)
521/nix/store/ifi6pp7q0ag5h7c5v9h1c1c7bhd10c7f-version
522
523$ cat /nix/store/ifi6pp7q0ag5h7c5v9h1c1c7bhd10c7f-version
52423.11
525Hello, world!
526```
527
528Note that the `hash` specified in the package is different than the hash specified in [](#ex-fetchers-fetchurl-nixpkgs-version), because the contents of the output have changed (even though the actual file that was downloaded is the same).
529See [](#chap-pkgs-fetchers-caveats) for more details on how to work with the `hash` attribute when the output changes.
530:::
531
532## `fetchzip` {#sec-pkgs-fetchers-fetchzip}
533
534Returns a [fixed-output derivation](https://nixos.org/manual/nix/stable/glossary.html#gloss-fixed-output-derivation) which downloads an archive from a given URL and decompresses it.
535
536Despite its name, `fetchzip` is not limited to `.zip` files but can also be used with [various compressed tarball formats](#tar-files) by default.
537This can extended by specifying additional attributes, see [](#ex-fetchers-fetchzip-rar-archive) to understand how to do that.
538
539### Inputs {#sec-pkgs-fetchers-fetchzip-inputs}
540
541`fetchzip` requires an attribute set, and most attributes are passed to the underlying call to [`fetchurl`](#sec-pkgs-fetchers-fetchurl).
542
543The attributes below are treated differently by `fetchzip` when compared to what `fetchurl` expects:
544
545`name` (String; _optional_)
546: Works as defined in `fetchurl`, but has a different default value than `fetchurl`.
547
548 _Default value:_ `"source"`.
549
550`nativeBuildInputs` (List of Attribute Set; _optional_)
551: Works as defined in `fetchurl`, but it is also augmented by `fetchzip` to include packages to deal with additional archives (such as `.zip`).
552
553 _Default value:_ `[]`.
554
555`postFetch` (String; _optional_)
556: Works as defined in `fetchurl`, but it is also augmented with the code needed to make `fetchzip` work.
557
558 :::{.caution}
559 It is only safe to modify files in `$out` in `postFetch`.
560 Consult the implementation of `fetchzip` for anything more involved.
561 :::
562
563 _Default value:_ `""`.
564
565`stripRoot` (Boolean; _optional_)
566: If `true`, the decompressed contents are moved one level up the directory tree.
567
568 This is useful for archives that decompress into a single directory which commonly includes some values that change with time, such as version numbers.
569 When this is the case (and `stripRoot` is `true`), `fetchzip` will remove this directory and make the decompressed contents available in the top-level directory.
570
571 [](#ex-fetchers-fetchzip-simple-striproot) shows what this attribute does.
572
573 This attribute is **not** passed through to `fetchurl`.
574
575 _Default value:_ `true`.
576
577`extension` (String or Null; _optional_)
578: If set, the archive downloaded by `fetchzip` will be renamed to a filename with the extension specified in this attribute.
579
580 This is useful when making `fetchzip` support additional types of archives, because the implementation may use the extension of an archive to determine whether they can decompress it.
581 If the URL you're using to download the contents doesn't end with the extension associated with the archive, use this attribute to fix the filename of the archive.
582
583 This attribute is **not** passed through to `fetchurl`.
584
585 _Default value:_ `null`.
586
587`recursiveHash` (Boolean; _optional_)
588: Works [as defined in `fetchurl`](#sec-pkgs-fetchers-fetchurl-inputs-recursiveHash), but its default value is different than for `fetchurl`.
589
590 _Default value:_ `true`.
591
592`downloadToTemp` (Boolean; _optional_)
593: Works [as defined in `fetchurl`](#sec-pkgs-fetchers-fetchurl-inputs-downloadToTemp), but its default value is different than for `fetchurl`.
594
595 _Default value:_ `true`.
596
597`extraPostFetch` **DEPRECATED**
598: This attribute is deprecated.
599 Please use `postFetch` instead.
600
601 This attribute is **not** passed through to `fetchurl`.
602
603### Examples {#sec-pkgs-fetchers-fetchzip-examples}
604
605::::{.example #ex-fetchers-fetchzip-simple-striproot}
606# Using `fetchzip` to output contents directly
607
608The following recipe shows how to use `fetchzip` to decompress a `.tar.gz` archive:
609
610```nix
611{ fetchzip }:
612fetchzip {
613 url = "https://github.com/NixOS/patchelf/releases/download/0.18.0/patchelf-0.18.0.tar.gz";
614 hash = "sha256-3ABYlME9R8klcpJ7MQpyFEFwHmxDDEzIYBqu/CpDYmg=";
615}
616```
617
618This archive has all its contents in a directory named `patchelf-0.18.0`.
619This means that after decompressing, you'd have to enter this directory to see the contents of the archive.
620However, `fetchzip` makes this easier through the attribute `stripRoot` (enabled by default).
621
622After building the recipe, the derivation output will show all the files in the archive at the top level:
623
624```shell
625$ nix-build
626(output removed for clarity)
627/nix/store/1b7h3fvmgrcddvs0m299hnqxlgli1yjw-source
628
629$ ls /nix/store/1b7h3fvmgrcddvs0m299hnqxlgli1yjw-source
630aclocal.m4 completions configure.ac m4 Makefile.in patchelf.spec README.md tests
631build-aux configure COPYING Makefile.am patchelf.1 patchelf.spec.in src version
632```
633
634If `stripRoot` is set to `false`, the derivation output will be the decompressed archive as-is:
635
636```nix
637{ fetchzip }:
638fetchzip {
639 url = "https://github.com/NixOS/patchelf/releases/download/0.18.0/patchelf-0.18.0.tar.gz";
640 hash = "sha256-uv3FuKE4DqpHT3yfE0qcnq0gYjDNQNKZEZt2+PUAneg=";
641 stripRoot = false;
642}
643```
644
645:::{.caution}
646The hash changed!
647Whenever changing attributes of a Nixpkgs fetcher, [remember to invalidate the hash](#chap-pkgs-fetchers-caveats), otherwise you won't get the results you're expecting!
648:::
649
650After building the recipe:
651
652```shell
653$ nix-build
654(output removed for clarity)
655/nix/store/2hy5bxw7xgbgxkn0i4x6hjr8w3dbx16c-source
656
657$ ls /nix/store/2hy5bxw7xgbgxkn0i4x6hjr8w3dbx16c-source
658patchelf-0.18.0
659```
660::::
661
662::::{.example #ex-fetchers-fetchzip-rar-archive}
663# Using `fetchzip` to decompress a `.rar` file
664
665The `unrar` package provides a [setup hook](#ssec-setup-hooks) to decompress `.rar` archives during the [unpack phase](#ssec-unpack-phase), which can be used with `fetchzip` to decompress those archives:
666
667```nix
668{ fetchzip, unrar }:
669fetchzip {
670 url = "https://archive.org/download/SpaceCadet_Plus95/Space_Cadet.rar";
671 hash = "sha256-fC+zsR8BY6vXpUkVd6i1jF0IZZxVKVvNi6VWCKT+pA4=";
672 stripRoot = false;
673 nativeBuildInputs = [ unrar ];
674}
675```
676
677Since this particular `.rar` file doesn't put its contents in a directory inside the archive, `stripRoot` must be set to `false`.
678
679After building the recipe, the derivation output will show the decompressed files:
680
681```shell
682$ nix-build
683(output removed for clarity)
684/nix/store/zpn7knxfva6rfjja2gbb4p3l9w1f0d36-source
685
686$ ls /nix/store/zpn7knxfva6rfjja2gbb4p3l9w1f0d36-source
687FONT.DAT PINBALL.DAT PINBALL.EXE PINBALL2.MID TABLE.BMP WMCONFIG.EXE
688MSCREATE.DIR PINBALL.DOC PINBALL.MID Sounds WAVEMIX.INF
689```
690::::
691
692## `fetchpatch` {#fetchpatch}
693
694`fetchpatch` works very similarly to `fetchurl` with the same arguments expected. It expects patch files as a source and performs normalization on them before computing the checksum. For example, it will remove comments or other unstable parts that are sometimes added by version control systems and can change over time.
695
696- `relative`: Similar to using `git-diff`'s `--relative` flag, only keep changes inside the specified directory, making paths relative to it.
697- `stripLen`: Remove the first `stripLen` components of pathnames in the patch.
698- `decode`: Pipe the downloaded data through this command before processing it as a patch.
699- `extraPrefix`: Prefix pathnames by this string.
700- `excludes`: Exclude files matching these patterns (applies after the above arguments).
701- `includes`: Include only files matching these patterns (applies after the above arguments).
702- `revert`: Revert the patch.
703
704Note that because the checksum is computed after applying these effects, using or modifying these arguments will have no effect unless the `hash` argument is changed as well.
705
706
707Most other fetchers return a directory rather than a single file.
708
709
710## `fetchDebianPatch` {#fetchdebianpatch}
711
712A wrapper around `fetchpatch`, which takes:
713- `patch` and `hash`: the patch's filename,
714 and its hash after normalization by `fetchpatch` ;
715- `pname`: the Debian source package's name ;
716- `version`: the upstream version number ;
717- `debianRevision`: the [Debian revision number] if applicable ;
718- the `area` of the Debian archive: `main` (default), `contrib`, or `non-free`.
719
720Here is an example of `fetchDebianPatch` in action:
721
722```nix
723{
724 lib,
725 fetchDebianPatch,
726 buildPythonPackage,
727}:
728
729buildPythonPackage rec {
730 pname = "pysimplesoap";
731 version = "1.16.2";
732 src = <...>;
733
734 patches = [
735 (fetchDebianPatch {
736 inherit pname version;
737 debianRevision = "5";
738 patch = "Add-quotes-to-SOAPAction-header-in-SoapClient.patch";
739 hash = "sha256-xA8Wnrpr31H8wy3zHSNfezFNjUJt1HbSXn3qUMzeKc0=";
740 })
741 ];
742
743 # ...
744}
745```
746
747Patches are fetched from `sources.debian.org`, and so must come from a
748package version that was uploaded to the Debian archive. Packages may
749be removed from there once that specific version isn't in any suite
750anymore (stable, testing, unstable, etc.), so maintainers should use
751`copy-tarballs.pl` to archive the patch if it needs to be available
752longer-term.
753
754[Debian revision number]: https://www.debian.org/doc/debian-policy/ch-controlfields.html#version
755
756
757## `fetchsvn` {#fetchsvn}
758
759Used with Subversion. Expects `url` to a Subversion directory, `rev`, and `hash`.
760
761## `fetchgit` {#fetchgit}
762
763Used with Git. Expects `url` to a Git repo, `rev`, and `hash`. `rev` in this case can be full the git commit id (SHA1 hash) or a tag name like `refs/tags/v1.0`.
764
765If you want to fetch a tag you should pass the `tag` parameter instead of `rev` which has the same effect as setting `rev = "refs/tags"/${version}"`.
766This is safer than just setting `rev = version` w.r.t. possible branch and tag name conflicts.
767
768Additionally, the following optional arguments can be given:
769
770*`fetchSubmodules`* (Boolean)
771
772: Whether to also fetch the submodules of a repository.
773
774*`fetchLFS`* (Boolean)
775
776: Whether to fetch LFS objects.
777
778*`preFetch`* (String)
779
780: Shell code to be executed before the repository has been fetched, to allow
781 changing the environment the fetcher runs in.
782
783*`postFetch`* (String)
784
785: Shell code executed after the repository has been fetched successfully.
786 This can do things like check or transform the file.
787
788*`leaveDotGit`* (Boolean)
789
790: Whether the `.git` directory of the clone should *not* be removed after checkout.
791
792 Be warned though that the git repository format is not stable and this flag is therefore not suitable for actual use by itself.
793 Only use this for testing purposes or in conjunction with removing the `.git` directory in `postFetch`.
794
795*`deepClone`* (Boolean)
796
797: Clone the entire repository as opposing to just creating a shallow clone.
798 This implies `leaveDotGit`.
799
800*`fetchTags`* (Boolean)
801
802: Whether to fetch all tags from the remote repository. This is useful when the build process needs to run `git describe` or other commands that require tag information to be available. This parameter implies `leaveDotGit`, as tags are stored in the `.git` directory.
803
804*`sparseCheckout`* (List of String)
805
806: Prevent git from fetching unnecessary blobs from server.
807 This is useful if only parts of the repository are needed.
808
809 ::: {.example #ex-fetchgit-sparseCheckout}
810
811 # Use `sparseCheckout` to only include some directories:
812
813 ```nix
814 { stdenv, fetchgit }:
815
816 stdenv.mkDerivation {
817 name = "hello";
818 src = fetchgit {
819 url = "https://...";
820 sparseCheckout = [
821 "directory/to/be/included"
822 "another/directory"
823 ];
824 hash = "sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=";
825 };
826 }
827 ```
828 :::
829
830 See [git sparse-checkout](https://git-scm.com/docs/git-sparse-checkout) for more information.
831
832*`rootDir`* (String)
833
834: When not empty, copy only contents of the subdirectory of the repository to the result. Automatically sets `sparseCheckout` and `nonConeMode` to avoid checking out any extra pieces. Incompatible with `leaveDotGit`.
835
836Some additional parameters for niche use-cases can be found listed in the function parameters in the declaration of `fetchgit`: `pkgs/build-support/fetchgit/default.nix`.
837Future parameters additions might also happen without immediately being documented here.
838
839## `fetchfossil` {#fetchfossil}
840
841Used with Fossil. Expects `url` to a Fossil archive, `rev`, and `hash`.
842
843## `fetchcvs` {#fetchcvs}
844
845Used with CVS. Expects `cvsRoot`, `tag`, and `hash`.
846
847## `fetchhg` {#fetchhg}
848
849Used with Mercurial. Expects `url`, `rev`, `hash`, overridable with [`<pkg>.overrideAttrs`](#sec-pkg-overrideAttrs).
850
851A number of fetcher functions wrap part of `fetchurl` and `fetchzip`. They are mainly convenience functions intended for commonly used destinations of source code in Nixpkgs. These wrapper fetchers are listed below.
852
853## `fetchFromGitea` {#fetchfromgitea}
854
855`fetchFromGitea` expects five arguments. `domain` is the gitea server name. `owner` is a string corresponding to the Gitea user or organization that controls this repository. `repo` corresponds to the name of the software repository. These are located at the top of every Gitea HTML page as `owner`/`repo`. `rev` corresponds to the Git commit hash or tag (e.g `v1.0`) that will be downloaded from Git. Finally, `hash` corresponds to the hash of the extracted directory. Again, other hash algorithms are also available but `hash` is currently preferred.
856
857## `fetchFromGitHub` {#fetchfromgithub}
858
859`fetchFromGitHub` expects four arguments. `owner` is a string corresponding to the GitHub user or organization that controls this repository. `repo` corresponds to the name of the software repository. These are located at the top of every GitHub HTML page as `owner`/`repo`. `rev` corresponds to the Git commit hash or tag (e.g `v1.0`) that will be downloaded from Git. If you need to fetch a tag however, you should prefer to use the `tag` parameter which achieves this in a safer way with less boilerplate. Finally, `hash` corresponds to the hash of the extracted directory. Again, other hash algorithms are also available, but `hash` is currently preferred.
860
861To use a different GitHub instance, use `githubBase` (defaults to `"github.com"`).
862
863`fetchFromGitHub` uses `fetchzip` to download the source archive generated by GitHub for the specified revision. If `leaveDotGit`, `deepClone` or `fetchSubmodules` are set to `true`, `fetchFromGitHub` will use `fetchgit` instead. Refer to its section for documentation of these options.
864
865## `fetchFromGitLab` {#fetchfromgitlab}
866
867This is used with GitLab repositories. It behaves similarly to `fetchFromGitHub`, and expects `owner`, `repo`, `rev`, and `hash`.
868
869To use a specific GitLab instance, use `domain` (defaults to `"gitlab.com"`).
870
871
872## `fetchFromGitiles` {#fetchfromgitiles}
873
874This is used with Gitiles repositories. The arguments expected are similar to `fetchgit`.
875
876## `fetchFromBitbucket` {#fetchfrombitbucket}
877
878This is used with BitBucket repositories. The arguments expected are very similar to `fetchFromGitHub` above.
879
880## `fetchFromSavannah` {#fetchfromsavannah}
881
882This is used with Savannah repositories. The arguments expected are very similar to `fetchFromGitHub` above.
883
884## `fetchFromRepoOrCz` {#fetchfromrepoorcz}
885
886This is used with repo.or.cz repositories. The arguments expected are very similar to `fetchFromGitHub` above.
887
888## `fetchFromSourcehut` {#fetchfromsourcehut}
889
890This is used with sourcehut repositories. Similar to `fetchFromGitHub` above,
891it expects `owner`, `repo`, `rev` and `hash`, but don't forget the tilde (~)
892in front of the username! Expected arguments also include `vc` ("git" (default)
893or "hg"), `domain` and `fetchSubmodules`.
894
895If `fetchSubmodules` is `true`, `fetchFromSourcehut` uses `fetchgit`
896or `fetchhg` with `fetchSubmodules` or `fetchSubrepos` set to `true`,
897respectively. Otherwise, the fetcher uses `fetchzip`.
898
899## `fetchFromRadicle` {#fetchfromradicle}
900
901This is used with Radicle repositories. The arguments expected are similar to `fetchgit`.
902
903Requires a `seed` argument (e.g. `seed.radicle.xyz` or `rosa.radicle.xyz`) and a `repo` argument
904(the repository id *without* the `rad:` prefix). Also accepts an optional `node` argument which
905contains the id of the node from which to fetch the specified ref. If `node` is `null` (the
906default), a canonical ref is fetched instead.
907
908```nix
909fetchFromRadicle {
910 seed = "seed.radicle.xyz";
911 repo = "z3gqcJUoA1n9HaHKufZs5FCSGazv5"; # heartwood
912 tag = "releases/1.3.0";
913 hash = "sha256-4o88BWKGGOjCIQy7anvzbA/kPOO+ZsLMzXJhE61odjw=";
914}
915```
916
917## `fetchRadiclePatch` {#fetchradiclepatch}
918
919`fetchRadiclePatch` works very similarly to `fetchFromRadicle` with almost the same arguments
920expected. However, instead of a `rev` or `tag` argument, a `revision` argument is expected, which
921contains the full revision id of the Radicle patch to fetch.
922
923```nix
924fetchRadiclePatch {
925 seed = "rosa.radicle.xyz";
926 repo = "z4V1sjrXqjvFdnCUbxPFqd5p4DtH5"; # radicle-explorer
927 revision = "d97d872386c70607beda2fb3fc2e60449e0f4ce4"; # patch: d77e064
928 hash = "sha256-ttnNqj0lhlSP6BGzEhhUOejKkkPruM9yMwA5p9Di4bk=";
929}
930```
931
932## `requireFile` {#requirefile}
933
934`requireFile` allows requesting files that cannot be fetched automatically, but whose content is known.
935This is a useful last-resort workaround for license restrictions that prohibit redistribution, or for downloads that are only accessible after authenticating interactively in a browser.
936If the requested file is present in the Nix store, the resulting derivation will not be built, because its expected output is already available.
937Otherwise, the builder will run, but fail with a message explaining to the user how to provide the file. The following code, for example:
938
939```nix
940requireFile {
941 name = "jdk-${version}_linux-x64_bin.tar.gz";
942 url = "https://www.oracle.com/java/technologies/javase-jdk11-downloads.html";
943 hash = "sha256-lL00+F7jjT71nlKJ7HRQuUQ7kkxVYlZh//5msD8sjeI=";
944}
945```
946results in this error message:
947```
948***
949Unfortunately, we cannot download file jdk-11.0.10_linux-x64_bin.tar.gz automatically.
950Please go to https://www.oracle.com/java/technologies/javase-jdk11-downloads.html to download it yourself, and add it to the Nix store
951using either
952 nix-store --add-fixed sha256 jdk-11.0.10_linux-x64_bin.tar.gz
953or
954 nix-prefetch-url --type sha256 file:///path/to/jdk-11.0.10_linux-x64_bin.tar.gz
955
956***
957```
958
959This function should only be used by non-redistributable software with an unfree license that we need to require the user to download manually.
960It produces packages that cannot be built automatically.
961
962## `fetchtorrent` {#fetchtorrent}
963
964`fetchtorrent` expects two arguments. `url` which can either be a Magnet URI (Magnet Link) such as `magnet:?xt=urn:btih:dd8255ecdc7ca55fb0bbf81323d87062db1f6d1c` or an HTTP URL pointing to a `.torrent` file. It can also take a `config` argument which will craft a `settings.json` configuration file and give it to `transmission`, the underlying program that is performing the fetch. The available config options for `transmission` can be found [here](https://github.com/transmission/transmission/blob/main/docs/Editing-Configuration-Files.md#options)
965
966```nix
967{ fetchtorrent }:
968
969fetchtorrent {
970 config = {
971 peer-limit-global = 100;
972 };
973 url = "magnet:?xt=urn:btih:dd8255ecdc7ca55fb0bbf81323d87062db1f6d1c";
974 hash = "";
975}
976```
977
978### Parameters {#fetchtorrent-parameters}
979
980- `url`: Magnet URI (Magnet Link) such as `magnet:?xt=urn:btih:dd8255ecdc7ca55fb0bbf81323d87062db1f6d1c` or an HTTP URL pointing to a `.torrent` file.
981
982- `backend`: Which bittorrent program to use. Default: `"transmission"`. Valid values are `"rqbit"` or `"transmission"`. These are the two most suitable torrent clients for fetching in a fixed-output derivation at the time of writing, as they can be easily exited after usage. `rqbit` is written in Rust and has a smaller closure size than `transmission`, and the performance and peer discovery properties differs between these clients, requiring experimentation to decide upon which is the best.
983
984- `config`: When using `transmission` as the `backend`, a json configuration can
985 be supplied to transmission. Refer to the [upstream documentation](https://github.com/transmission/transmission/blob/main/docs/Editing-Configuration-Files.md) for information on how to configure.
986