pyrox.dev
1<chapter xmlns="http://docbook.org/ns/docbook"
2 xmlns:xlink="http://www.w3.org/1999/xlink"
3 xml:id="chap-functions">
4
5<title>Functions reference</title>
6
7<para>
8 The nixpkgs repository has several utility functions to manipulate Nix expressions.
9</para>
10
11<section xml:id="sec-overrides">
12 <title>Overriding</title>
13
14 <para>
15 Sometimes one wants to override parts of
16 <literal>nixpkgs</literal>, e.g. derivation attributes, the results of
17 derivations or even the whole package set.
18 </para>
19
20 <section xml:id="sec-pkg-override">
21 <title><pkg>.override</title>
22
23 <para>
24 The function <varname>override</varname> is usually available for all the
25 derivations in the nixpkgs expression (<varname>pkgs</varname>).
26 </para>
27 <para>
28 It is used to override the arguments passed to a function.
29 </para>
30 <para>
31 Example usages:
32
33 <programlisting>pkgs.foo.override { arg1 = val1; arg2 = val2; ... }</programlisting>
34 <programlisting>import pkgs.path { overlays = [ (self: super: {
35 foo = super.foo.override { barSupport = true ; };
36 })]};</programlisting>
37 <programlisting>mypkg = pkgs.callPackage ./mypkg.nix {
38 mydep = pkgs.mydep.override { ... };
39 }</programlisting>
40 </para>
41
42 <para>
43 In the first example, <varname>pkgs.foo</varname> is the result of a function call
44 with some default arguments, usually a derivation.
45 Using <varname>pkgs.foo.override</varname> will call the same function with
46 the given new arguments.
47 </para>
48
49 </section>
50
51 <section xml:id="sec-pkg-overrideAttrs">
52 <title><pkg>.overrideAttrs</title>
53
54 <para>
55 The function <varname>overrideAttrs</varname> allows overriding the
56 attribute set passed to a <varname>stdenv.mkDerivation</varname> call,
57 producing a new derivation based on the original one.
58 This function is available on all derivations produced by the
59 <varname>stdenv.mkDerivation</varname> function, which is most packages
60 in the nixpkgs expression <varname>pkgs</varname>.
61 </para>
62
63 <para>
64 Example usage:
65
66 <programlisting>helloWithDebug = pkgs.hello.overrideAttrs (oldAttrs: rec {
67 separateDebugInfo = true;
68 });</programlisting>
69 </para>
70
71 <para>
72 In the above example, the <varname>separateDebugInfo</varname> attribute is
73 overridden to be true, thus building debug info for
74 <varname>helloWithDebug</varname>, while all other attributes will be
75 retained from the original <varname>hello</varname> package.
76 </para>
77
78 <para>
79 The argument <varname>oldAttrs</varname> is conventionally used to refer to
80 the attr set originally passed to <varname>stdenv.mkDerivation</varname>.
81 </para>
82
83 <note>
84 <para>
85 Note that <varname>separateDebugInfo</varname> is processed only by the
86 <varname>stdenv.mkDerivation</varname> function, not the generated, raw
87 Nix derivation. Thus, using <varname>overrideDerivation</varname> will
88 not work in this case, as it overrides only the attributes of the final
89 derivation. It is for this reason that <varname>overrideAttrs</varname>
90 should be preferred in (almost) all cases to
91 <varname>overrideDerivation</varname>, i.e. to allow using
92 <varname>sdenv.mkDerivation</varname> to process input arguments, as well
93 as the fact that it is easier to use (you can use the same attribute
94 names you see in your Nix code, instead of the ones generated (e.g.
95 <varname>buildInputs</varname> vs <varname>nativeBuildInputs</varname>,
96 and involves less typing.
97 </para>
98 </note>
99
100 </section>
101
102
103 <section xml:id="sec-pkg-overrideDerivation">
104 <title><pkg>.overrideDerivation</title>
105
106 <warning>
107 <para>You should prefer <varname>overrideAttrs</varname> in almost all
108 cases, see its documentation for the reasons why.
109 <varname>overrideDerivation</varname> is not deprecated and will continue
110 to work, but is less nice to use and does not have as many abilities as
111 <varname>overrideAttrs</varname>.
112 </para>
113 </warning>
114
115 <warning>
116 <para>Do not use this function in Nixpkgs as it evaluates a Derivation
117 before modifying it, which breaks package abstraction and removes
118 error-checking of function arguments. In addition, this
119 evaluation-per-function application incurs a performance penalty,
120 which can become a problem if many overrides are used.
121 It is only intended for ad-hoc customisation, such as in
122 <filename>~/.config/nixpkgs/config.nix</filename>.
123 </para>
124 </warning>
125
126 <para>
127 The function <varname>overrideDerivation</varname> creates a new derivation
128 based on an existing one by overriding the original's attributes with
129 the attribute set produced by the specified function.
130 This function is available on all
131 derivations defined using the <varname>makeOverridable</varname> function.
132 Most standard derivation-producing functions, such as
133 <varname>stdenv.mkDerivation</varname>, are defined using this
134 function, which means most packages in the nixpkgs expression,
135 <varname>pkgs</varname>, have this function.
136 </para>
137
138 <para>
139 Example usage:
140
141 <programlisting>mySed = pkgs.gnused.overrideDerivation (oldAttrs: {
142 name = "sed-4.2.2-pre";
143 src = fetchurl {
144 url = ftp://alpha.gnu.org/gnu/sed/sed-4.2.2-pre.tar.bz2;
145 sha256 = "11nq06d131y4wmf3drm0yk502d2xc6n5qy82cg88rb9nqd2lj41k";
146 };
147 patches = [];
148 });</programlisting>
149 </para>
150
151 <para>
152 In the above example, the <varname>name</varname>, <varname>src</varname>,
153 and <varname>patches</varname> of the derivation will be overridden, while
154 all other attributes will be retained from the original derivation.
155 </para>
156
157 <para>
158 The argument <varname>oldAttrs</varname> is used to refer to the attribute set of
159 the original derivation.
160 </para>
161
162 <note>
163 <para>
164 A package's attributes are evaluated *before* being modified by
165 the <varname>overrideDerivation</varname> function.
166 For example, the <varname>name</varname> attribute reference
167 in <varname>url = "mirror://gnu/hello/${name}.tar.gz";</varname>
168 is filled-in *before* the <varname>overrideDerivation</varname> function
169 modifies the attribute set. This means that overriding the
170 <varname>name</varname> attribute, in this example, *will not* change the
171 value of the <varname>url</varname> attribute. Instead, we need to override
172 both the <varname>name</varname> *and* <varname>url</varname> attributes.
173 </para>
174 </note>
175
176 </section>
177
178 <section xml:id="sec-lib-makeOverridable">
179 <title>lib.makeOverridable</title>
180
181 <para>
182 The function <varname>lib.makeOverridable</varname> is used to make the result
183 of a function easily customizable. This utility only makes sense for functions
184 that accept an argument set and return an attribute set.
185 </para>
186
187 <para>
188 Example usage:
189
190 <programlisting>f = { a, b }: { result = a+b; }
191 c = lib.makeOverridable f { a = 1; b = 2; }</programlisting>
192
193 </para>
194
195 <para>
196 The variable <varname>c</varname> is the value of the <varname>f</varname> function
197 applied with some default arguments. Hence the value of <varname>c.result</varname>
198 is <literal>3</literal>, in this example.
199 </para>
200
201 <para>
202 The variable <varname>c</varname> however also has some additional functions, like
203 <link linkend="sec-pkg-override">c.override</link> which can be used to
204 override the default arguments. In this example the value of
205 <varname>(c.override { a = 4; }).result</varname> is 6.
206 </para>
207
208 </section>
209
210</section>
211
212<section xml:id="sec-generators">
213 <title>Generators</title>
214
215 <para>
216 Generators are functions that create file formats from nix
217 data structures, e. g. for configuration files.
218 There are generators available for: <literal>INI</literal>,
219 <literal>JSON</literal> and <literal>YAML</literal>
220 </para>
221
222 <para>
223 All generators follow a similar call interface: <code>generatorName
224 configFunctions data</code>, where <literal>configFunctions</literal> is a
225 set of user-defined functions that format variable parts of the content.
226 They each have common defaults, so often they do not need to be set
227 manually. An example is <code>mkSectionName ? (name: libStr.escape [ "[" "]"
228 ] name)</code> from the <literal>INI</literal> generator. It gets the name
229 of a section and returns a sanitized name. The default
230 <literal>mkSectionName</literal> escapes <literal>[</literal> and
231 <literal>]</literal> with a backslash.
232 </para>
233
234 <note><para>Nix store paths can be converted to strings by enclosing a
235 derivation attribute like so: <code>"${drv}"</code>.</para></note>
236
237 <para>
238 Detailed documentation for each generator can be found in
239 <literal>lib/generators.nix</literal>.
240 </para>
241
242</section>
243
244
245<section xml:id="sec-fhs-environments">
246 <title>buildFHSUserEnv</title>
247
248 <para>
249 <function>buildFHSUserEnv</function> provides a way to build and run
250 FHS-compatible lightweight sandboxes. It creates an isolated root with
251 bound <filename>/nix/store</filename>, so its footprint in terms of disk
252 space needed is quite small. This allows one to run software which is hard or
253 unfeasible to patch for NixOS -- 3rd-party source trees with FHS assumptions,
254 games distributed as tarballs, software with integrity checking and/or external
255 self-updated binaries. It uses Linux namespaces feature to create
256 temporary lightweight environments which are destroyed after all child
257 processes exit, without root user rights requirement. Accepted arguments are:
258 </para>
259
260 <variablelist>
261 <varlistentry>
262 <term><literal>name</literal></term>
263
264 <listitem><para>Environment name.</para></listitem>
265 </varlistentry>
266
267 <varlistentry>
268 <term><literal>targetPkgs</literal></term>
269
270 <listitem><para>Packages to be installed for the main host's architecture
271 (i.e. x86_64 on x86_64 installations). Along with libraries binaries are also
272 installed.</para></listitem>
273 </varlistentry>
274
275 <varlistentry>
276 <term><literal>multiPkgs</literal></term>
277
278 <listitem><para>Packages to be installed for all architectures supported by
279 a host (i.e. i686 and x86_64 on x86_64 installations). Only libraries are
280 installed by default.</para></listitem>
281 </varlistentry>
282
283 <varlistentry>
284 <term><literal>extraBuildCommands</literal></term>
285
286 <listitem><para>Additional commands to be executed for finalizing the
287 directory structure.</para></listitem>
288 </varlistentry>
289
290 <varlistentry>
291 <term><literal>extraBuildCommandsMulti</literal></term>
292
293 <listitem><para>Like <literal>extraBuildCommands</literal>, but
294 executed only on multilib architectures.</para></listitem>
295 </varlistentry>
296
297 <varlistentry>
298 <term><literal>extraOutputsToInstall</literal></term>
299
300 <listitem><para>Additional derivation outputs to be linked for both
301 target and multi-architecture packages.</para></listitem>
302 </varlistentry>
303
304 <varlistentry>
305 <term><literal>extraInstallCommands</literal></term>
306
307 <listitem><para>Additional commands to be executed for finalizing the
308 derivation with runner script.</para></listitem>
309 </varlistentry>
310
311 <varlistentry>
312 <term><literal>runScript</literal></term>
313
314 <listitem><para>A command that would be executed inside the sandbox and
315 passed all the command line arguments. It defaults to
316 <literal>bash</literal>.</para></listitem>
317 </varlistentry>
318 </variablelist>
319
320 <para>
321 One can create a simple environment using a <literal>shell.nix</literal>
322 like that:
323 </para>
324
325<programlisting><![CDATA[
326{ pkgs ? import <nixpkgs> {} }:
327
328(pkgs.buildFHSUserEnv {
329 name = "simple-x11-env";
330 targetPkgs = pkgs: (with pkgs;
331 [ udev
332 alsaLib
333 ]) ++ (with pkgs.xorg;
334 [ libX11
335 libXcursor
336 libXrandr
337 ]);
338 multiPkgs = pkgs: (with pkgs;
339 [ udev
340 alsaLib
341 ]);
342 runScript = "bash";
343}).env
344]]></programlisting>
345
346 <para>
347 Running <literal>nix-shell</literal> would then drop you into a shell with
348 these libraries and binaries available. You can use this to run
349 closed-source applications which expect FHS structure without hassles:
350 simply change <literal>runScript</literal> to the application path,
351 e.g. <filename>./bin/start.sh</filename> -- relative paths are supported.
352 </para>
353</section>
354
355<section xml:id="sec-pkgs-dockerTools">
356<title>pkgs.dockerTools</title>
357
358<para>
359 <varname>pkgs.dockerTools</varname> is a set of functions for creating and
360 manipulating Docker images according to the
361 <link xlink:href="https://github.com/moby/moby/blob/master/image/spec/v1.2.md#docker-image-specification-v120">
362 Docker Image Specification v1.2.0
363 </link>. Docker itself is not used to perform any of the operations done by these
364 functions.
365</para>
366
367<warning>
368 <para>
369 The <varname>dockerTools</varname> API is unstable and may be subject to
370 backwards-incompatible changes in the future.
371 </para>
372</warning>
373
374<section xml:id="ssec-pkgs-dockerTools-buildImage">
375 <title>buildImage</title>
376
377 <para>
378 This function is analogous to the <command>docker build</command> command,
379 in that can used to build a Docker-compatible repository tarball containing
380 a single image with one or multiple layers. As such, the result
381 is suitable for being loaded in Docker with <command>docker load</command>.
382 </para>
383
384 <para>
385 The parameters of <varname>buildImage</varname> with relative example values are
386 described below:
387 </para>
388
389 <example xml:id='ex-dockerTools-buildImage'><title>Docker build</title>
390 <programlisting>
391 buildImage {
392 name = "redis"; <co xml:id='ex-dockerTools-buildImage-1' />
393 tag = "latest"; <co xml:id='ex-dockerTools-buildImage-2' />
394
395 fromImage = someBaseImage; <co xml:id='ex-dockerTools-buildImage-3' />
396 fromImageName = null; <co xml:id='ex-dockerTools-buildImage-4' />
397 fromImageTag = "latest"; <co xml:id='ex-dockerTools-buildImage-5' />
398
399 contents = pkgs.redis; <co xml:id='ex-dockerTools-buildImage-6' />
400 runAsRoot = '' <co xml:id='ex-dockerTools-buildImage-runAsRoot' />
401 #!${stdenv.shell}
402 mkdir -p /data
403 '';
404
405 config = { <co xml:id='ex-dockerTools-buildImage-8' />
406 Cmd = [ "/bin/redis-server" ];
407 WorkingDir = "/data";
408 Volumes = {
409 "/data" = {};
410 };
411 };
412 }
413 </programlisting>
414 </example>
415
416 <para>The above example will build a Docker image <literal>redis/latest</literal>
417 from the given base image. Loading and running this image in Docker results in
418 <literal>redis-server</literal> being started automatically.
419 </para>
420
421 <calloutlist>
422 <callout arearefs='ex-dockerTools-buildImage-1'>
423 <para>
424 <varname>name</varname> specifies the name of the resulting image.
425 This is the only required argument for <varname>buildImage</varname>.
426 </para>
427 </callout>
428
429 <callout arearefs='ex-dockerTools-buildImage-2'>
430 <para>
431 <varname>tag</varname> specifies the tag of the resulting image.
432 By default it's <literal>latest</literal>.
433 </para>
434 </callout>
435
436 <callout arearefs='ex-dockerTools-buildImage-3'>
437 <para>
438 <varname>fromImage</varname> is the repository tarball containing the base image.
439 It must be a valid Docker image, such as exported by <command>docker save</command>.
440 By default it's <literal>null</literal>, which can be seen as equivalent
441 to <literal>FROM scratch</literal> of a <filename>Dockerfile</filename>.
442 </para>
443 </callout>
444
445 <callout arearefs='ex-dockerTools-buildImage-4'>
446 <para>
447 <varname>fromImageName</varname> can be used to further specify
448 the base image within the repository, in case it contains multiple images.
449 By default it's <literal>null</literal>, in which case
450 <varname>buildImage</varname> will peek the first image available
451 in the repository.
452 </para>
453 </callout>
454
455 <callout arearefs='ex-dockerTools-buildImage-5'>
456 <para>
457 <varname>fromImageTag</varname> can be used to further specify the tag
458 of the base image within the repository, in case an image contains multiple tags.
459 By default it's <literal>null</literal>, in which case
460 <varname>buildImage</varname> will peek the first tag available for the base image.
461 </para>
462 </callout>
463
464 <callout arearefs='ex-dockerTools-buildImage-6'>
465 <para>
466 <varname>contents</varname> is a derivation that will be copied in the new
467 layer of the resulting image. This can be similarly seen as
468 <command>ADD contents/ /</command> in a <filename>Dockerfile</filename>.
469 By default it's <literal>null</literal>.
470 </para>
471 </callout>
472
473 <callout arearefs='ex-dockerTools-buildImage-runAsRoot'>
474 <para>
475 <varname>runAsRoot</varname> is a bash script that will run as root
476 in an environment that overlays the existing layers of the base image with
477 the new resulting layer, including the previously copied
478 <varname>contents</varname> derivation.
479 This can be similarly seen as
480 <command>RUN ...</command> in a <filename>Dockerfile</filename>.
481
482 <note>
483 <para>
484 Using this parameter requires the <literal>kvm</literal>
485 device to be available.
486 </para>
487 </note>
488 </para>
489 </callout>
490
491 <callout arearefs='ex-dockerTools-buildImage-8'>
492 <para>
493 <varname>config</varname> is used to specify the configuration of the
494 containers that will be started off the built image in Docker.
495 The available options are listed in the
496 <link xlink:href="https://github.com/moby/moby/blob/master/image/spec/v1.2.md#image-json-field-descriptions">
497 Docker Image Specification v1.2.0
498 </link>.
499 </para>
500 </callout>
501
502 </calloutlist>
503
504 <para>
505 After the new layer has been created, its closure
506 (to which <varname>contents</varname>, <varname>config</varname> and
507 <varname>runAsRoot</varname> contribute) will be copied in the layer itself.
508 Only new dependencies that are not already in the existing layers will be copied.
509 </para>
510
511 <para>
512 At the end of the process, only one new single layer will be produced and
513 added to the resulting image.
514 </para>
515
516 <para>
517 The resulting repository will only list the single image
518 <varname>image/tag</varname>. In the case of <xref linkend='ex-dockerTools-buildImage'/>
519 it would be <varname>redis/latest</varname>.
520 </para>
521
522 <para>
523 It is possible to inspect the arguments with which an image was built
524 using its <varname>buildArgs</varname> attribute.
525 </para>
526
527
528
529 <note>
530 <para>
531 If you see errors similar to <literal>getProtocolByName: does not exist (no such protocol name: tcp)</literal>
532 you may need to add <literal>pkgs.iana-etc</literal> to <varname>contents</varname>.
533 </para>
534 </note>
535
536 <note>
537 <para>
538 If you see errors similar to <literal>Error_Protocol ("certificate has unknown CA",True,UnknownCa)</literal>
539 you may need to add <literal>pkgs.cacert</literal> to <varname>contents</varname>.
540 </para>
541 </note>
542
543</section>
544
545<section xml:id="ssec-pkgs-dockerTools-fetchFromRegistry">
546 <title>pullImage</title>
547
548 <para>
549 This function is analogous to the <command>docker pull</command> command,
550 in that can be used to fetch a Docker image from a Docker registry.
551 Currently only registry <literal>v1</literal> is supported.
552 By default <link xlink:href="https://hub.docker.com/">Docker Hub</link>
553 is used to pull images.
554 </para>
555
556 <para>
557 Its parameters are described in the example below:
558 </para>
559
560 <example xml:id='ex-dockerTools-pullImage'><title>Docker pull</title>
561 <programlisting>
562 pullImage {
563 imageName = "debian"; <co xml:id='ex-dockerTools-pullImage-1' />
564 imageTag = "jessie"; <co xml:id='ex-dockerTools-pullImage-2' />
565 imageId = null; <co xml:id='ex-dockerTools-pullImage-3' />
566 sha256 = "1bhw5hkz6chrnrih0ymjbmn69hyfriza2lr550xyvpdrnbzr4gk2"; <co xml:id='ex-dockerTools-pullImage-4' />
567
568 indexUrl = "https://index.docker.io"; <co xml:id='ex-dockerTools-pullImage-5' />
569 registryVersion = "v1";
570 }
571 </programlisting>
572 </example>
573
574 <calloutlist>
575 <callout arearefs='ex-dockerTools-pullImage-1'>
576 <para>
577 <varname>imageName</varname> specifies the name of the image to be downloaded,
578 which can also include the registry namespace (e.g. <literal>library/debian</literal>).
579 This argument is required.
580 </para>
581 </callout>
582
583 <callout arearefs='ex-dockerTools-pullImage-2'>
584 <para>
585 <varname>imageTag</varname> specifies the tag of the image to be downloaded.
586 By default it's <literal>latest</literal>.
587 </para>
588 </callout>
589
590 <callout arearefs='ex-dockerTools-pullImage-3'>
591 <para>
592 <varname>imageId</varname>, if specified this exact image will be fetched, instead
593 of <varname>imageName/imageTag</varname>. However, the resulting repository
594 will still be named <varname>imageName/imageTag</varname>.
595 By default it's <literal>null</literal>.
596 </para>
597 </callout>
598
599 <callout arearefs='ex-dockerTools-pullImage-4'>
600 <para>
601 <varname>sha256</varname> is the checksum of the whole fetched image.
602 This argument is required.
603 </para>
604
605 <note>
606 <para>The checksum is computed on the unpacked directory, not on the final tarball.</para>
607 </note>
608
609 </callout>
610
611 <callout arearefs='ex-dockerTools-pullImage-5'>
612 <para>
613 In the above example the default values are shown for the variables
614 <varname>indexUrl</varname> and <varname>registryVersion</varname>.
615 Hence by default the Docker.io registry is used to pull the images.
616 </para>
617 </callout>
618 </calloutlist>
619
620</section>
621
622<section xml:id="ssec-pkgs-dockerTools-exportImage">
623 <title>exportImage</title>
624
625 <para>
626 This function is analogous to the <command>docker export</command> command,
627 in that can used to flatten a Docker image that contains multiple layers.
628 It is in fact the result of the merge of all the layers of the image.
629 As such, the result is suitable for being imported in Docker
630 with <command>docker import</command>.
631 </para>
632
633 <note>
634 <para>
635 Using this function requires the <literal>kvm</literal>
636 device to be available.
637 </para>
638 </note>
639
640 <para>
641 The parameters of <varname>exportImage</varname> are the following:
642 </para>
643
644 <example xml:id='ex-dockerTools-exportImage'><title>Docker export</title>
645 <programlisting>
646 exportImage {
647 fromImage = someLayeredImage;
648 fromImageName = null;
649 fromImageTag = null;
650
651 name = someLayeredImage.name;
652 }
653 </programlisting>
654 </example>
655
656 <para>
657 The parameters relative to the base image have the same synopsis as
658 described in <xref linkend='ssec-pkgs-dockerTools-buildImage'/>, except that
659 <varname>fromImage</varname> is the only required argument in this case.
660 </para>
661
662 <para>
663 The <varname>name</varname> argument is the name of the derivation output,
664 which defaults to <varname>fromImage.name</varname>.
665 </para>
666</section>
667
668<section xml:id="ssec-pkgs-dockerTools-shadowSetup">
669 <title>shadowSetup</title>
670
671 <para>
672 This constant string is a helper for setting up the base files for managing
673 users and groups, only if such files don't exist already.
674 It is suitable for being used in a
675 <varname>runAsRoot</varname> <xref linkend='ex-dockerTools-buildImage-runAsRoot'/> script for cases like
676 in the example below:
677 </para>
678
679 <example xml:id='ex-dockerTools-shadowSetup'><title>Shadow base files</title>
680 <programlisting>
681 buildImage {
682 name = "shadow-basic";
683
684 runAsRoot = ''
685 #!${stdenv.shell}
686 ${shadowSetup}
687 groupadd -r redis
688 useradd -r -g redis redis
689 mkdir /data
690 chown redis:redis /data
691 '';
692 }
693 </programlisting>
694 </example>
695
696 <para>
697 Creating base files like <literal>/etc/passwd</literal> or
698 <literal>/etc/login.defs</literal> are necessary for shadow-utils to
699 manipulate users and groups.
700 </para>
701
702</section>
703
704</section>
705
706</chapter>