pyrox.dev
1# Fetchers {#chap-pkgs-fetchers}
2
3Building software with Nix often requires downloading source code and other files from the internet.
4To this end, Nixpkgs provides *fetchers*: functions to obtain remote sources via various protocols and services.
5
6Nixpkgs fetchers differ from built-in fetchers such as [`builtins.fetchTarball`](https://nixos.org/manual/nix/stable/language/builtins.html#builtins-fetchTarball):
7- A built-in fetcher will download and cache files at evaluation time and produce a [store path](https://nixos.org/manual/nix/stable/glossary#gloss-store-path).
8 A Nixpkgs fetcher will create a ([fixed-output](https://nixos.org/manual/nix/stable/glossary#gloss-fixed-output-derivation)) [derivation](https://nixos.org/manual/nix/stable/language/derivations), and files are downloaded at build time.
9- Built-in fetchers will invalidate their cache after [`tarball-ttl`](https://nixos.org/manual/nix/stable/command-ref/conf-file#conf-tarball-ttl) expires, and will require network activity to check if the cache entry is up to date.
10 Nixpkgs fetchers only re-download if the specified hash changes or the store object is not otherwise available.
11- Built-in fetchers do not use [substituters](https://nixos.org/manual/nix/stable/command-ref/conf-file#conf-substituters).
12 Derivations produced by Nixpkgs fetchers will use any configured binary cache transparently.
13
14This significantly reduces the time needed to evaluate the entirety of Nixpkgs, and allows [Hydra](https://nixos.org/hydra) to retain and re-distribute sources used by Nixpkgs in the [public binary cache](https://cache.nixos.org).
15For these reasons, built-in fetchers are not allowed in Nixpkgs source code.
16
17The following table shows an overview of the differences:
18
19| Fetchers | Download | Output | Cache | Re-download when |
20|-|-|-|-|-|
21| `builtins.fetch*` | evaluation time | store path | `/nix/store`, `~/.cache/nix` | `tarball-ttl` expires, cache miss in `~/.cache/nix`, output store object not in local store |
22| `pkgs.fetch*` | build time | derivation | `/nix/store`, substituters | output store object not available |
23
24## Caveats {#chap-pkgs-fetchers-caveats}
25
26The fact that the hash belongs to the Nix derivation output and not the file itself can lead to confusion.
27For example, consider the following fetcher:
28
29```nix
30fetchurl {
31 url = "http://www.example.org/hello-1.0.tar.gz";
32 hash = "sha256-lTeyxzJNQeMdu1IVdovNMtgn77jRIhSybLdMbTkf2Ww=";
33};
34```
35
36A common mistake is to update a fetcher’s URL, or a version parameter, without updating the hash.
37
38```nix
39fetchurl {
40 url = "http://www.example.org/hello-1.1.tar.gz";
41 hash = "sha256-lTeyxzJNQeMdu1IVdovNMtgn77jRIhSybLdMbTkf2Ww=";
42};
43```
44
45**This will reuse the old contents**.
46Remember to invalidate the hash argument, in this case by setting the `hash` attribute to an empty string.
47
48```nix
49fetchurl {
50 url = "http://www.example.org/hello-1.1.tar.gz";
51 hash = "";
52};
53```
54
55Use the resulting error message to determine the correct hash.
56
57```
58error: hash mismatch in fixed-output derivation '/path/to/my.drv':
59 specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
60 got: sha256-lTeyxzJNQeMdu1IVdovNMtgn77jRIhSybLdMbTkf2Ww=
61```
62
63A similar problem arises while testing changes to a fetcher's implementation. If the output of the derivation already exists in the Nix store, test failures can go undetected. The [`invalidateFetcherByDrvHash`](#tester-invalidateFetcherByDrvHash) function helps prevent reusing cached derivations.
64
65## `fetchurl` and `fetchzip` {#fetchurl}
66
67Two basic fetchers are `fetchurl` and `fetchzip`. Both of these have two required arguments, a URL and a hash. The hash is typically `hash`, although many more hash algorithms are supported. Nixpkgs contributors are currently recommended to use `hash`. This hash will be used by Nix to identify your source. A typical usage of `fetchurl` is provided below.
68
69```nix
70{ stdenv, fetchurl }:
71
72stdenv.mkDerivation {
73 name = "hello";
74 src = fetchurl {
75 url = "http://www.example.org/hello.tar.gz";
76 hash = "sha256-BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB=";
77 };
78}
79```
80
81The main difference between `fetchurl` and `fetchzip` is in how they store the contents. `fetchurl` will store the unaltered contents of the URL within the Nix store. `fetchzip` on the other hand, will decompress the archive for you, making files and directories directly accessible in the future. `fetchzip` can only be used with archives. Despite the name, `fetchzip` is not limited to .zip files and can also be used with any tarball.
82
83## `fetchpatch` {#fetchpatch}
84
85`fetchpatch` works very similarly to `fetchurl` with the same arguments expected. It expects patch files as a source and performs normalization on them before computing the checksum. For example, it will remove comments or other unstable parts that are sometimes added by version control systems and can change over time.
86
87- `relative`: Similar to using `git-diff`'s `--relative` flag, only keep changes inside the specified directory, making paths relative to it.
88- `stripLen`: Remove the first `stripLen` components of pathnames in the patch.
89- `decode`: Pipe the downloaded data through this command before processing it as a patch.
90- `extraPrefix`: Prefix pathnames by this string.
91- `excludes`: Exclude files matching these patterns (applies after the above arguments).
92- `includes`: Include only files matching these patterns (applies after the above arguments).
93- `revert`: Revert the patch.
94
95Note that because the checksum is computed after applying these effects, using or modifying these arguments will have no effect unless the `hash` argument is changed as well.
96
97
98Most other fetchers return a directory rather than a single file.
99
100
101## `fetchDebianPatch` {#fetchdebianpatch}
102
103A wrapper around `fetchpatch`, which takes:
104- `patch` and `hash`: the patch's filename,
105 and its hash after normalization by `fetchpatch` ;
106- `pname`: the Debian source package's name ;
107- `version`: the upstream version number ;
108- `debianRevision`: the [Debian revision number] if applicable ;
109- the `area` of the Debian archive: `main` (default), `contrib`, or `non-free`.
110
111Here is an example of `fetchDebianPatch` in action:
112
113```nix
114{ lib
115, fetchDebianPatch
116, buildPythonPackage
117}:
118
119buildPythonPackage rec {
120 pname = "pysimplesoap";
121 version = "1.16.2";
122 src = ...;
123
124 patches = [
125 (fetchDebianPatch {
126 inherit pname version;
127 debianRevision = "5";
128 name = "Add-quotes-to-SOAPAction-header-in-SoapClient.patch";
129 hash = "sha256-xA8Wnrpr31H8wy3zHSNfezFNjUJt1HbSXn3qUMzeKc0=";
130 })
131 ];
132
133 ...
134}
135```
136
137Patches are fetched from `sources.debian.org`, and so must come from a
138package version that was uploaded to the Debian archive. Packages may
139be removed from there once that specific version isn't in any suite
140anymore (stable, testing, unstable, etc.), so maintainers should use
141`copy-tarballs.pl` to archive the patch if it needs to be available
142longer-term.
143
144[Debian revision number]: https://www.debian.org/doc/debian-policy/ch-controlfields.html#version
145
146
147## `fetchsvn` {#fetchsvn}
148
149Used with Subversion. Expects `url` to a Subversion directory, `rev`, and `hash`.
150
151## `fetchgit` {#fetchgit}
152
153Used with Git. Expects `url` to a Git repo, `rev`, and `hash`. `rev` in this case can be full the git commit id (SHA1 hash) or a tag name like `refs/tags/v1.0`.
154
155Additionally, the following optional arguments can be given: `fetchSubmodules = true` makes `fetchgit` also fetch the submodules of a repository. If `deepClone` is set to true, the entire repository is cloned as opposing to just creating a shallow clone. `deepClone = true` also implies `leaveDotGit = true` which means that the `.git` directory of the clone won't be removed after checkout.
156
157If only parts of the repository are needed, `sparseCheckout` can be used. This will prevent git from fetching unnecessary blobs from server, see [git sparse-checkout](https://git-scm.com/docs/git-sparse-checkout) for more information:
158
159```nix
160{ stdenv, fetchgit }:
161
162stdenv.mkDerivation {
163 name = "hello";
164 src = fetchgit {
165 url = "https://...";
166 sparseCheckout = [
167 "directory/to/be/included"
168 "another/directory"
169 ];
170 hash = "sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=";
171 };
172}
173```
174
175## `fetchfossil` {#fetchfossil}
176
177Used with Fossil. Expects `url` to a Fossil archive, `rev`, and `hash`.
178
179## `fetchcvs` {#fetchcvs}
180
181Used with CVS. Expects `cvsRoot`, `tag`, and `hash`.
182
183## `fetchhg` {#fetchhg}
184
185Used with Mercurial. Expects `url`, `rev`, and `hash`.
186
187A number of fetcher functions wrap part of `fetchurl` and `fetchzip`. They are mainly convenience functions intended for commonly used destinations of source code in Nixpkgs. These wrapper fetchers are listed below.
188
189## `fetchFromGitea` {#fetchfromgitea}
190
191`fetchFromGitea` expects five arguments. `domain` is the gitea server name. `owner` is a string corresponding to the Gitea user or organization that controls this repository. `repo` corresponds to the name of the software repository. These are located at the top of every Gitea HTML page as `owner`/`repo`. `rev` corresponds to the Git commit hash or tag (e.g `v1.0`) that will be downloaded from Git. Finally, `hash` corresponds to the hash of the extracted directory. Again, other hash algorithms are also available but `hash` is currently preferred.
192
193## `fetchFromGitHub` {#fetchfromgithub}
194
195`fetchFromGitHub` expects four arguments. `owner` is a string corresponding to the GitHub user or organization that controls this repository. `repo` corresponds to the name of the software repository. These are located at the top of every GitHub HTML page as `owner`/`repo`. `rev` corresponds to the Git commit hash or tag (e.g `v1.0`) that will be downloaded from Git. Finally, `hash` corresponds to the hash of the extracted directory. Again, other hash algorithms are also available, but `hash` is currently preferred.
196
197To use a different GitHub instance, use `githubBase` (defaults to `"github.com"`).
198
199`fetchFromGitHub` uses `fetchzip` to download the source archive generated by GitHub for the specified revision. If `leaveDotGit`, `deepClone` or `fetchSubmodules` are set to `true`, `fetchFromGitHub` will use `fetchgit` instead. Refer to its section for documentation of these options.
200
201## `fetchFromGitLab` {#fetchfromgitlab}
202
203This is used with GitLab repositories. It behaves similarly to `fetchFromGitHub`, and expects `owner`, `repo`, `rev`, and `hash`.
204
205To use a specific GitLab instance, use `domain` (defaults to `"gitlab.com"`).
206
207
208## `fetchFromGitiles` {#fetchfromgitiles}
209
210This is used with Gitiles repositories. The arguments expected are similar to `fetchgit`.
211
212## `fetchFromBitbucket` {#fetchfrombitbucket}
213
214This is used with BitBucket repositories. The arguments expected are very similar to `fetchFromGitHub` above.
215
216## `fetchFromSavannah` {#fetchfromsavannah}
217
218This is used with Savannah repositories. The arguments expected are very similar to `fetchFromGitHub` above.
219
220## `fetchFromRepoOrCz` {#fetchfromrepoorcz}
221
222This is used with repo.or.cz repositories. The arguments expected are very similar to `fetchFromGitHub` above.
223
224## `fetchFromSourcehut` {#fetchfromsourcehut}
225
226This is used with sourcehut repositories. Similar to `fetchFromGitHub` above,
227it expects `owner`, `repo`, `rev` and `hash`, but don't forget the tilde (~)
228in front of the username! Expected arguments also include `vc` ("git" (default)
229or "hg"), `domain` and `fetchSubmodules`.
230
231If `fetchSubmodules` is `true`, `fetchFromSourcehut` uses `fetchgit`
232or `fetchhg` with `fetchSubmodules` or `fetchSubrepos` set to `true`,
233respectively. Otherwise, the fetcher uses `fetchzip`.
234
235## `requireFile` {#requirefile}
236
237`requireFile` allows requesting files that cannot be fetched automatically, but whose content is known.
238This is a useful last-resort workaround for license restrictions that prohibit redistribution, or for downloads that are only accessible after authenticating interactively in a browser.
239If the requested file is present in the Nix store, the resulting derivation will not be built, because its expected output is already available.
240Otherwise, the builder will run, but fail with a message explaining to the user how to provide the file. The following code, for example:
241
242```
243requireFile {
244 name = "jdk-${version}_linux-x64_bin.tar.gz";
245 url = "https://www.oracle.com/java/technologies/javase-jdk11-downloads.html";
246 hash = "sha256-lL00+F7jjT71nlKJ7HRQuUQ7kkxVYlZh//5msD8sjeI=";
247}
248```
249results in this error message:
250```
251***
252Unfortunately, we cannot download file jdk-11.0.10_linux-x64_bin.tar.gz automatically.
253Please go to https://www.oracle.com/java/technologies/javase-jdk11-downloads.html to download it yourself, and add it to the Nix store
254using either
255 nix-store --add-fixed sha256 jdk-11.0.10_linux-x64_bin.tar.gz
256or
257 nix-prefetch-url --type sha256 file:///path/to/jdk-11.0.10_linux-x64_bin.tar.gz
258
259***
260```
261## `fetchtorrent` {#fetchtorrent}
262
263`fetchtorrent` expects two arguments. `url` which can either be a Magnet URI (Magnet Link) such as `magnet:?xt=urn:btih:dd8255ecdc7ca55fb0bbf81323d87062db1f6d1c` or an HTTP URL pointing to a `.torrent` file. It can also take a `config` argument which will craft a `settings.json` configuration file and give it to `transmission`, the underlying program that is performing the fetch. The available config options for `transmission` can be found [here](https://github.com/transmission/transmission/blob/main/docs/Editing-Configuration-Files.md#options)
264
265```
266{ fetchtorrent }:
267
268fetchtorrent {
269 config = { peer-limit-global = 100; };
270 url = "magnet:?xt=urn:btih:dd8255ecdc7ca55fb0bbf81323d87062db1f6d1c";
271 sha256 = "";
272}
273```
274
275### Parameters {#fetchtorrent-parameters}
276
277- `url`: Magnet URI (Magnet Link) such as `magnet:?xt=urn:btih:dd8255ecdc7ca55fb0bbf81323d87062db1f6d1c` or an HTTP URL pointing to a `.torrent` file.
278
279- `backend`: Which bittorrent program to use. Default: `"transmission"`. Valid values are `"rqbit"` or `"transmission"`. These are the two most suitable torrent clients for fetching in a fixed-output derivation at the time of writing, as they can be easily exited after usage. `rqbit` is written in Rust and has a smaller closure size than `transmission`, and the performance and peer discovery properties differs between these clients, requiring experimentation to decide upon which is the best.
280
281- `config`: When using `transmission` as the `backend`, a json configuration can
282 be supplied to transmission. Refer to the [upstream documentation](https://github.com/transmission/transmission/blob/main/docs/Editing-Configuration-Files.md) for information on how to configure.
283