pyrox.dev
1# pkgs.dockerTools {#sec-pkgs-dockerTools}
2
3`pkgs.dockerTools` is a set of functions for creating and manipulating Docker images according to the [Docker Image Specification v1.2.0](https://github.com/moby/moby/blob/master/image/spec/v1.2.md#docker-image-specification-v120). Docker itself is not used to perform any of the operations done by these functions.
4
5## buildImage {#ssec-pkgs-dockerTools-buildImage}
6
7This function is analogous to the `docker build` command, in that it can be used to build a Docker-compatible repository tarball containing a single image with one or multiple layers. As such, the result is suitable for being loaded in Docker with `docker load`.
8
9The parameters of `buildImage` with relative example values are described below:
10
11[]{#ex-dockerTools-buildImage}
12[]{#ex-dockerTools-buildImage-runAsRoot}
13
14```nix
15buildImage {
16 name = "redis";
17 tag = "latest";
18
19 fromImage = someBaseImage;
20 fromImageName = null;
21 fromImageTag = "latest";
22
23 copyToRoot = pkgs.buildEnv {
24 name = "image-root";
25 paths = [ pkgs.redis ];
26 pathsToLink = [ "/bin" ];
27 };
28
29 runAsRoot = ''
30 #!${pkgs.runtimeShell}
31 mkdir -p /data
32 '';
33
34 config = {
35 Cmd = [ "/bin/redis-server" ];
36 WorkingDir = "/data";
37 Volumes = { "/data" = { }; };
38 };
39
40 diskSize = 1024;
41 buildVMMemorySize = 512;
42}
43```
44
45The above example will build a Docker image `redis/latest` from the given base image. Loading and running this image in Docker results in `redis-server` being started automatically.
46
47- `name` specifies the name of the resulting image. This is the only required argument for `buildImage`.
48
49- `tag` specifies the tag of the resulting image. By default it's `null`, which indicates that the nix output hash will be used as tag.
50
51- `fromImage` is the repository tarball containing the base image. It must be a valid Docker image, such as exported by `docker save`. By default it's `null`, which can be seen as equivalent to `FROM scratch` of a `Dockerfile`.
52
53- `fromImageName` can be used to further specify the base image within the repository, in case it contains multiple images. By default it's `null`, in which case `buildImage` will peek the first image available in the repository.
54
55- `fromImageTag` can be used to further specify the tag of the base image within the repository, in case an image contains multiple tags. By default it's `null`, in which case `buildImage` will peek the first tag available for the base image.
56
57- `copyToRoot` is a derivation that will be copied in the new layer of the resulting image. This can be similarly seen as `ADD contents/ /` in a `Dockerfile`. By default it's `null`.
58
59- `runAsRoot` is a bash script that will run as root in an environment that overlays the existing layers of the base image with the new resulting layer, including the previously copied `contents` derivation. This can be similarly seen as `RUN ...` in a `Dockerfile`.
60
61> **_NOTE:_** Using this parameter requires the `kvm` device to be available.
62
63- `config` is used to specify the configuration of the containers that will be started off the built image in Docker. The available options are listed in the [Docker Image Specification v1.2.0](https://github.com/moby/moby/blob/master/image/spec/v1.2.md#image-json-field-descriptions).
64
65- `architecture` is _optional_ and used to specify the image architecture, this is useful for multi-architecture builds that don't need cross compiling. If not specified it will default to `hostPlatform`.
66
67- `diskSize` is used to specify the disk size of the VM used to build the image in megabytes. By default it's 1024 MiB.
68
69- `buildVMMemorySize` is used to specify the memory size of the VM to build the image in megabytes. By default it's 512 MiB.
70
71After the new layer has been created, its closure (to which `contents`, `config` and `runAsRoot` contribute) will be copied in the layer itself. Only new dependencies that are not already in the existing layers will be copied.
72
73At the end of the process, only one new single layer will be produced and added to the resulting image.
74
75The resulting repository will only list the single image `image/tag`. In the case of [the `buildImage` example](#ex-dockerTools-buildImage), it would be `redis/latest`.
76
77It is possible to inspect the arguments with which an image was built using its `buildArgs` attribute.
78
79> **_NOTE:_** If you see errors similar to `getProtocolByName: does not exist (no such protocol name: tcp)` you may need to add `pkgs.iana-etc` to `contents`.
80
81> **_NOTE:_** If you see errors similar to `Error_Protocol ("certificate has unknown CA",True,UnknownCa)` you may need to add `pkgs.cacert` to `contents`.
82
83By default `buildImage` will use a static date of one second past the UNIX Epoch. This allows `buildImage` to produce binary reproducible images. When listing images with `docker images`, the newly created images will be listed like this:
84
85```ShellSession
86$ docker images
87REPOSITORY TAG IMAGE ID CREATED SIZE
88hello latest 08c791c7846e 48 years ago 25.2MB
89```
90
91You can break binary reproducibility but have a sorted, meaningful `CREATED` column by setting `created` to `now`.
92
93```nix
94pkgs.dockerTools.buildImage {
95 name = "hello";
96 tag = "latest";
97 created = "now";
98 copyToRoot = pkgs.buildEnv {
99 name = "image-root";
100 paths = [ pkgs.hello ];
101 pathsToLink = [ "/bin" ];
102 };
103
104 config.Cmd = [ "/bin/hello" ];
105}
106```
107
108Now the Docker CLI will display a reasonable date and sort the images as expected:
109
110```ShellSession
111$ docker images
112REPOSITORY TAG IMAGE ID CREATED SIZE
113hello latest de2bf4786de6 About a minute ago 25.2MB
114```
115
116However, the produced images will not be binary reproducible.
117
118## buildLayeredImage {#ssec-pkgs-dockerTools-buildLayeredImage}
119
120Create a Docker image with many of the store paths being on their own layer to improve sharing between images. The image is realized into the Nix store as a gzipped tarball. Depending on the intended usage, many users might prefer to use `streamLayeredImage` instead, which this function uses internally.
121
122`name`
123
124: The name of the resulting image.
125
126`tag` _optional_
127
128: Tag of the generated image.
129
130 *Default:* the output path's hash
131
132`fromImage` _optional_
133
134: The repository tarball containing the base image. It must be a valid Docker image, such as one exported by `docker save`.
135
136 *Default:* `null`, which can be seen as equivalent to `FROM scratch` of a `Dockerfile`.
137
138`contents` _optional_
139
140: Top-level paths in the container. Either a single derivation, or a list of derivations.
141
142 *Default:* `[]`
143
144`config` _optional_
145
146`architecture` is _optional_ and used to specify the image architecture, this is useful for multi-architecture builds that don't need cross compiling. If not specified it will default to `hostPlatform`.
147
148: Run-time configuration of the container. A full list of the options available is in the [Docker Image Specification v1.2.0](https://github.com/moby/moby/blob/master/image/spec/v1.2.md#image-json-field-descriptions).
149
150 *Default:* `{}`
151
152`created` _optional_
153
154: Date and time the layers were created. Follows the same `now` exception supported by `buildImage`.
155
156 *Default:* `1970-01-01T00:00:01Z`
157
158`maxLayers` _optional_
159
160: Maximum number of layers to create.
161
162 *Default:* `100`
163
164 *Maximum:* `125`
165
166`extraCommands` _optional_
167
168: Shell commands to run while building the final layer, without access to most of the layer contents. Changes to this layer are "on top" of all the other layers, so can create additional directories and files.
169
170`fakeRootCommands` _optional_
171
172: Shell commands to run while creating the archive for the final layer in a fakeroot environment. Unlike `extraCommands`, you can run `chown` to change the owners of the files in the archive, changing fakeroot's state instead of the real filesystem. The latter would require privileges that the build user does not have. Static binaries do not interact with the fakeroot environment. By default all files in the archive will be owned by root.
173
174`enableFakechroot` _optional_
175
176: Whether to run in `fakeRootCommands` in `fakechroot`, making programs behave as though `/` is the root of the image being created, while files in the Nix store are available as usual. This allows scripts that perform installation in `/` to work as expected. Considering that `fakechroot` is implemented via the same mechanism as `fakeroot`, the same caveats apply.
177
178 *Default:* `false`
179
180### Behavior of `contents` in the final image {#dockerTools-buildLayeredImage-arg-contents}
181
182Each path directly listed in `contents` will have a symlink in the root of the image.
183
184For example:
185
186```nix
187pkgs.dockerTools.buildLayeredImage {
188 name = "hello";
189 contents = [ pkgs.hello ];
190}
191```
192
193will create symlinks for all the paths in the `hello` package:
194
195```ShellSession
196/bin/hello -> /nix/store/h1zb1padqbbb7jicsvkmrym3r6snphxg-hello-2.10/bin/hello
197/share/info/hello.info -> /nix/store/h1zb1padqbbb7jicsvkmrym3r6snphxg-hello-2.10/share/info/hello.info
198/share/locale/bg/LC_MESSAGES/hello.mo -> /nix/store/h1zb1padqbbb7jicsvkmrym3r6snphxg-hello-2.10/share/locale/bg/LC_MESSAGES/hello.mo
199```
200
201### Automatic inclusion of `config` references {#dockerTools-buildLayeredImage-arg-config}
202
203The closure of `config` is automatically included in the closure of the final image.
204
205This allows you to make very simple Docker images with very little code. This container will start up and run `hello`:
206
207```nix
208pkgs.dockerTools.buildLayeredImage {
209 name = "hello";
210 config.Cmd = [ "${pkgs.hello}/bin/hello" ];
211}
212```
213
214### Adjusting `maxLayers` {#dockerTools-buildLayeredImage-arg-maxLayers}
215
216Increasing the `maxLayers` increases the number of layers which have a chance to be shared between different images.
217
218Modern Docker installations support up to 128 layers, but older versions support as few as 42.
219
220If the produced image will not be extended by other Docker builds, it is safe to set `maxLayers` to `128`. However, it will be impossible to extend the image further.
221
222The first (`maxLayers-2`) most "popular" paths will have their own individual layers, then layer \#`maxLayers-1` will contain all the remaining "unpopular" paths, and finally layer \#`maxLayers` will contain the Image configuration.
223
224Docker's Layers are not inherently ordered, they are content-addressable and are not explicitly layered until they are composed in to an Image.
225
226## streamLayeredImage {#ssec-pkgs-dockerTools-streamLayeredImage}
227
228Builds a script which, when run, will stream an uncompressed tarball of a Docker image to stdout. The arguments to this function are as for `buildLayeredImage`. This method of constructing an image does not realize the image into the Nix store, so it saves on IO and disk/cache space, particularly with large images.
229
230The image produced by running the output script can be piped directly into `docker load`, to load it into the local docker daemon:
231
232```ShellSession
233$(nix-build) | docker load
234```
235
236Alternatively, the image be piped via `gzip` into `skopeo`, e.g., to copy it into a registry:
237
238```ShellSession
239$(nix-build) | gzip --fast | skopeo copy docker-archive:/dev/stdin docker://some_docker_registry/myimage:tag
240```
241
242## pullImage {#ssec-pkgs-dockerTools-fetchFromRegistry}
243
244This function is analogous to the `docker pull` command, in that it can be used to pull a Docker image from a Docker registry. By default [Docker Hub](https://hub.docker.com/) is used to pull images.
245
246Its parameters are described in the example below:
247
248```nix
249pullImage {
250 imageName = "nixos/nix";
251 imageDigest =
252 "sha256:473a2b527958665554806aea24d0131bacec46d23af09fef4598eeab331850fa";
253 finalImageName = "nix";
254 finalImageTag = "2.11.1";
255 sha256 = "sha256-qvhj+Hlmviz+KEBVmsyPIzTB3QlVAFzwAY1zDPIBGxc=";
256 os = "linux";
257 arch = "x86_64";
258}
259```
260
261- `imageName` specifies the name of the image to be downloaded, which can also include the registry namespace (e.g. `nixos`). This argument is required.
262
263- `imageDigest` specifies the digest of the image to be downloaded. This argument is required.
264
265- `finalImageName`, if specified, this is the name of the image to be created. Note it is never used to fetch the image since we prefer to rely on the immutable digest ID. By default it's equal to `imageName`.
266
267- `finalImageTag`, if specified, this is the tag of the image to be created. Note it is never used to fetch the image since we prefer to rely on the immutable digest ID. By default it's `latest`.
268
269- `sha256` is the checksum of the whole fetched image. This argument is required.
270
271- `os`, if specified, is the operating system of the fetched image. By default it's `linux`.
272
273- `arch`, if specified, is the cpu architecture of the fetched image. By default it's `x86_64`.
274
275`nix-prefetch-docker` command can be used to get required image parameters:
276
277```ShellSession
278$ nix run nixpkgs#nix-prefetch-docker -- --image-name mysql --image-tag 5
279```
280
281Since a given `imageName` may transparently refer to a manifest list of images which support multiple architectures and/or operating systems, you can supply the `--os` and `--arch` arguments to specify exactly which image you want. By default it will match the OS and architecture of the host the command is run on.
282
283```ShellSession
284$ nix-prefetch-docker --image-name mysql --image-tag 5 --arch x86_64 --os linux
285```
286
287Desired image name and tag can be set using `--final-image-name` and `--final-image-tag` arguments:
288
289```ShellSession
290$ nix-prefetch-docker --image-name mysql --image-tag 5 --final-image-name eu.gcr.io/my-project/mysql --final-image-tag prod
291```
292
293## exportImage {#ssec-pkgs-dockerTools-exportImage}
294
295This function is analogous to the `docker export` command, in that it can be used to flatten a Docker image that contains multiple layers. It is in fact the result of the merge of all the layers of the image. As such, the result is suitable for being imported in Docker with `docker import`.
296
297> **_NOTE:_** Using this function requires the `kvm` device to be available.
298
299The parameters of `exportImage` are the following:
300
301```nix
302exportImage {
303 fromImage = someLayeredImage;
304 fromImageName = null;
305 fromImageTag = null;
306
307 name = someLayeredImage.name;
308}
309```
310
311The parameters relative to the base image have the same synopsis as described in [buildImage](#ssec-pkgs-dockerTools-buildImage), except that `fromImage` is the only required argument in this case.
312
313The `name` argument is the name of the derivation output, which defaults to `fromImage.name`.
314
315## Environment Helpers {#ssec-pkgs-dockerTools-helpers}
316
317Some packages expect certain files to be available globally.
318When building an image from scratch (i.e. without `fromImage`), these files are missing.
319`pkgs.dockerTools` provides some helpers to set up an environment with the necessary files.
320You can include them in `copyToRoot` like this:
321
322```nix
323buildImage {
324 name = "environment-example";
325 copyToRoot = with pkgs.dockerTools; [
326 usrBinEnv
327 binSh
328 caCertificates
329 fakeNss
330 ];
331}
332```
333
334### usrBinEnv {#sssec-pkgs-dockerTools-helpers-usrBinEnv}
335
336This provides the `env` utility at `/usr/bin/env`.
337
338### binSh {#sssec-pkgs-dockerTools-helpers-binSh}
339
340This provides `bashInteractive` at `/bin/sh`.
341
342### caCertificates {#sssec-pkgs-dockerTools-helpers-caCertificates}
343
344This sets up `/etc/ssl/certs/ca-certificates.crt`.
345
346### fakeNss {#sssec-pkgs-dockerTools-helpers-fakeNss}
347
348Provides `/etc/passwd` and `/etc/group` that contain root and nobody.
349Useful when packaging binaries that insist on using nss to look up
350username/groups (like nginx).
351
352### shadowSetup {#ssec-pkgs-dockerTools-shadowSetup}
353
354This constant string is a helper for setting up the base files for managing users and groups, only if such files don't exist already. It is suitable for being used in a [`buildImage` `runAsRoot`](#ex-dockerTools-buildImage-runAsRoot) script for cases like in the example below:
355
356```nix
357buildImage {
358 name = "shadow-basic";
359
360 runAsRoot = ''
361 #!${pkgs.runtimeShell}
362 ${pkgs.dockerTools.shadowSetup}
363 groupadd -r redis
364 useradd -r -g redis redis
365 mkdir /data
366 chown redis:redis /data
367 '';
368}
369```
370
371Creating base files like `/etc/passwd` or `/etc/login.defs` is necessary for shadow-utils to manipulate users and groups.
372
373## fakeNss {#ssec-pkgs-dockerTools-fakeNss}
374
375If your primary goal is providing a basic skeleton for user lookups to work,
376and/or a lesser privileged user, adding `pkgs.fakeNss` to
377the container image root might be the better choice than a custom script
378running `useradd` and friends.
379
380It provides a `/etc/passwd` and `/etc/group`, containing `root` and `nobody`
381users and groups.
382
383It also provides a `/etc/nsswitch.conf`, configuring NSS host resolution to
384first check `/etc/hosts`, before checking DNS, as the default in the absence of
385a config file (`dns [!UNAVAIL=return] files`) is quite unexpected.
386
387You can pair it with `binSh`, which provides `bin/sh` as a symlink
388to `bashInteractive` (as `/bin/sh` is configured as a shell).
389
390```nix
391buildImage {
392 name = "shadow-basic";
393
394 copyToRoot = pkgs.buildEnv {
395 name = "image-root";
396 paths = [ binSh pkgs.fakeNss ];
397 pathsToLink = [ "/bin" "/etc" "/var" ];
398 };
399}
400```
401
402## buildNixShellImage {#ssec-pkgs-dockerTools-buildNixShellImage}
403
404Create a Docker image that sets up an environment similar to that of running `nix-shell` on a derivation.
405When run in Docker, this environment somewhat resembles the Nix sandbox typically used by `nix-build`, with a major difference being that access to the internet is allowed.
406It additionally also behaves like an interactive `nix-shell`, running things like `shellHook` and setting an interactive prompt.
407If the derivation is fully buildable (i.e. `nix-build` can be used on it), running `buildDerivation` inside such a Docker image will build the derivation, with all its outputs being available in the correct `/nix/store` paths, pointed to by the respective environment variables like `$out`, etc.
408
409::: {.warning}
410The behavior doesn't match `nix-shell` or `nix-build` exactly and this function is known not to work correctly for e.g. fixed-output derivations, content-addressed derivations, impure derivations and other special types of derivations.
411:::
412
413### Arguments {#ssec-pkgs-dockerTools-buildNixShellImage-arguments}
414
415`drv`
416
417: The derivation on which to base the Docker image.
418
419 Adding packages to the Docker image is possible by e.g. extending the list of `nativeBuildInputs` of this derivation like
420
421 ```nix
422 buildNixShellImage {
423 drv = someDrv.overrideAttrs (old: {
424 nativeBuildInputs = old.nativeBuildInputs or [] ++ [
425 somethingExtra
426 ];
427 });
428 # ...
429 }
430 ```
431
432 Similarly, you can extend the image initialization script by extending `shellHook`
433
434`name` _optional_
435
436: The name of the resulting image.
437
438 *Default:* `drv.name + "-env"`
439
440`tag` _optional_
441
442: Tag of the generated image.
443
444 *Default:* the resulting image derivation output path's hash
445
446`uid`/`gid` _optional_
447
448: The user/group ID to run the container as. This is like a `nixbld` build user.
449
450 *Default:* 1000/1000
451
452`homeDirectory` _optional_
453
454: The home directory of the user the container is running as
455
456 *Default:* `/build`
457
458`shell` _optional_
459
460: The path to the `bash` binary to use as the shell. This shell is started when running the image.
461
462 *Default:* `pkgs.bashInteractive + "/bin/bash"`
463
464`command` _optional_
465
466: Run this command in the environment of the derivation, in an interactive shell. See the `--command` option in the [`nix-shell` documentation](https://nixos.org/manual/nix/stable/command-ref/nix-shell.html?highlight=nix-shell#options).
467
468 *Default:* (none)
469
470`run` _optional_
471
472: Same as `command`, but runs the command in a non-interactive shell instead. See the `--run` option in the [`nix-shell` documentation](https://nixos.org/manual/nix/stable/command-ref/nix-shell.html?highlight=nix-shell#options).
473
474 *Default:* (none)
475
476### Example {#ssec-pkgs-dockerTools-buildNixShellImage-example}
477
478The following shows how to build the `pkgs.hello` package inside a Docker container built with `buildNixShellImage`.
479
480```nix
481with import <nixpkgs> {};
482dockerTools.buildNixShellImage {
483 drv = hello;
484}
485```
486
487Build the derivation:
488
489```console
490nix-build hello.nix
491```
492
493 these 8 derivations will be built:
494 /nix/store/xmw3a5ln29rdalavcxk1w3m4zb2n7kk6-nix-shell-rc.drv
495 ...
496 Creating layer 56 from paths: ['/nix/store/crpnj8ssz0va2q0p5ibv9i6k6n52gcya-stdenv-linux']
497 Creating layer 57 with customisation...
498 Adding manifests...
499 Done.
500 /nix/store/cpyn1lc897ghx0rhr2xy49jvyn52bazv-hello-2.12-env.tar.gz
501
502Load the image:
503
504```console
505docker load -i result
506```
507
508 0d9f4c4cd109: Loading layer [==================================================>] 2.56MB/2.56MB
509 ...
510 ab1d897c0697: Loading layer [==================================================>] 10.24kB/10.24kB
511 Loaded image: hello-2.12-env:pgj9h98nal555415faa43vsydg161bdz
512
513Run the container:
514
515```console
516docker run -it hello-2.12-env:pgj9h98nal555415faa43vsydg161bdz
517```
518
519 [nix-shell:/build]$
520
521In the running container, run the build:
522
523```console
524buildDerivation
525```
526
527 unpacking sources
528 unpacking source archive /nix/store/8nqv6kshb3vs5q5bs2k600xpj5bkavkc-hello-2.12.tar.gz
529 ...
530 patching script interpreter paths in /nix/store/z5wwy5nagzy15gag42vv61c2agdpz2f2-hello-2.12
531 checking for references to /build/ in /nix/store/z5wwy5nagzy15gag42vv61c2agdpz2f2-hello-2.12...
532
533Check the build result:
534
535```console
536$out/bin/hello
537```
538
539 Hello, world!