pyrox.dev
1<?xml version="1.0" encoding="UTF-8"?>
2<!DOCTYPE chapter [
3 <!ENTITY ndash "–"> <!-- @vcunat likes to use this one ;-) -->
4]>
5<chapter xmlns="http://docbook.org/ns/docbook"
6 xmlns:xlink="http://www.w3.org/1999/xlink"
7 xml:id="chap-multiple-output">
8 <title>Multiple-output packages</title>
9 <section xml:id="sec-multiple-outputs-introduction">
10 <title>Introduction</title>
11
12 <para>
13 The Nix language allows a derivation to produce multiple outputs, which is
14 similar to what is utilized by other Linux distribution packaging systems.
15 The outputs reside in separate nix store paths, so they can be mostly
16 handled independently of each other, including passing to build inputs,
17 garbage collection or binary substitution. The exception is that building
18 from source always produces all the outputs.
19 </para>
20
21 <para>
22 The main motivation is to save disk space by reducing runtime closure sizes;
23 consequently also sizes of substituted binaries get reduced. Splitting can
24 be used to have more granular runtime dependencies, for example the typical
25 reduction is to split away development-only files, as those are typically
26 not needed during runtime. As a result, closure sizes of many packages can
27 get reduced to a half or even much less.
28 </para>
29
30 <note>
31 <para>
32 The reduction effects could be instead achieved by building the parts in
33 completely separate derivations. That would often additionally reduce
34 build-time closures, but it tends to be much harder to write such
35 derivations, as build systems typically assume all parts are being built at
36 once. This compromise approach of single source package producing multiple
37 binary packages is also utilized often by rpm and deb.
38 </para>
39 </note>
40 </section>
41 <section xml:id="sec-multiple-outputs-installing">
42 <title>Installing a split package</title>
43
44 <para>
45 When installing a package via <varname>systemPackages</varname> or
46 <command>nix-env</command> you have several options:
47 </para>
48
49 <itemizedlist>
50 <listitem>
51 <para>
52 You can install particular outputs explicitly, as each is available in the
53 Nix language as an attribute of the package. The
54 <varname>outputs</varname> attribute contains a list of output names.
55 </para>
56 </listitem>
57 <listitem>
58 <para>
59 You can let it use the default outputs. These are handled by
60 <varname>meta.outputsToInstall</varname> attribute that contains a list of
61 output names.
62 </para>
63 <para>
64 TODO: more about tweaking the attribute, etc.
65 </para>
66 </listitem>
67 <listitem>
68 <para>
69 NixOS provides configuration option
70 <varname>environment.extraOutputsToInstall</varname> that allows adding
71 extra outputs of <varname>environment.systemPackages</varname> atop the
72 default ones. It's mainly meant for documentation and debug symbols, and
73 it's also modified by specific options.
74 </para>
75 <note>
76 <para>
77 At this moment there is no similar configurability for packages installed
78 by <command>nix-env</command>. You can still use approach from
79 <xref linkend="sec-modify-via-packageOverrides" /> to override
80 <varname>meta.outputsToInstall</varname> attributes, but that's a rather
81 inconvenient way.
82 </para>
83 </note>
84 </listitem>
85 </itemizedlist>
86 </section>
87 <section xml:id="sec-multiple-outputs-using-split-packages">
88 <title>Using a split package</title>
89
90 <para>
91 In the Nix language the individual outputs can be reached explicitly as
92 attributes, e.g. <varname>coreutils.info</varname>, but the typical case is
93 just using packages as build inputs.
94 </para>
95
96 <para>
97 When a multiple-output derivation gets into a build input of another
98 derivation, the <varname>dev</varname> output is added if it exists,
99 otherwise the first output is added. In addition to that,
100 <varname>propagatedBuildOutputs</varname> of that package which by default
101 contain <varname>$outputBin</varname> and <varname>$outputLib</varname> are
102 also added. (See <xref linkend="multiple-output-file-type-groups" />.)
103 </para>
104 </section>
105 <section xml:id="sec-multiple-outputs-">
106 <title>Writing a split derivation</title>
107
108 <para>
109 Here you find how to write a derivation that produces multiple outputs.
110 </para>
111
112 <para>
113 In nixpkgs there is a framework supporting multiple-output derivations. It
114 tries to cover most cases by default behavior. You can find the source
115 separated in
116 <<filename>nixpkgs/pkgs/build-support/setup-hooks/multiple-outputs.sh</filename>>;
117 it's relatively well-readable. The whole machinery is triggered by defining
118 the <varname>outputs</varname> attribute to contain the list of desired
119 output names (strings).
120 </para>
121
122<programlisting>outputs = [ "bin" "dev" "out" "doc" ];</programlisting>
123
124 <para>
125 Often such a single line is enough. For each output an equally named
126 environment variable is passed to the builder and contains the path in nix
127 store for that output. Typically you also want to have the main
128 <varname>out</varname> output, as it catches any files that didn't get
129 elsewhere.
130 </para>
131
132 <note>
133 <para>
134 There is a special handling of the <varname>debug</varname> output,
135 described at <xref linkend="stdenv-separateDebugInfo" />.
136 </para>
137 </note>
138
139 <section xml:id="multiple-output-file-binaries-first-convention">
140 <title><quote>Binaries first</quote></title>
141
142 <para>
143 A commonly adopted convention in <literal>nixpkgs</literal> is that
144 executables provided by the package are contained within its first output.
145 This convention allows the dependent packages to reference the executables
146 provided by packages in a uniform manner. For instance, provided with the
147 knowledge that the <literal>perl</literal> package contains a
148 <literal>perl</literal> executable it can be referenced as
149 <literal>${pkgs.perl}/bin/perl</literal> within a Nix derivation that needs
150 to execute a Perl script.
151 </para>
152
153 <para>
154 The <literal>glibc</literal> package is a deliberate single exception to
155 the <quote>binaries first</quote> convention. The <literal>glibc</literal>
156 has <literal>libs</literal> as its first output allowing the libraries
157 provided by <literal>glibc</literal> to be referenced directly (e.g.
158 <literal>${stdenv.glibc}/lib/ld-linux-x86-64.so.2</literal>). The
159 executables provided by <literal>glibc</literal> can be accessed via its
160 <literal>bin</literal> attribute (e.g.
161 <literal>${stdenv.glibc.bin}/bin/ldd</literal>).
162 </para>
163
164 <para>
165 The reason for why <literal>glibc</literal> deviates from the convention is
166 because referencing a library provided by <literal>glibc</literal> is a
167 very common operation among Nix packages. For instance, third-party
168 executables packaged by Nix are typically patched and relinked with the
169 relevant version of <literal>glibc</literal> libraries from Nix packages
170 (please see the documentation on
171 <link xlink:href="https://nixos.org/patchelf.html">patchelf</link> for more
172 details).
173 </para>
174 </section>
175
176 <section xml:id="multiple-output-file-type-groups">
177 <title>File type groups</title>
178
179 <para>
180 The support code currently recognizes some particular kinds of outputs and
181 either instructs the build system of the package to put files into their
182 desired outputs or it moves the files during the fixup phase. Each group of
183 file types has an <varname>outputFoo</varname> variable specifying the
184 output name where they should go. If that variable isn't defined by the
185 derivation writer, it is guessed – a default output name is defined,
186 falling back to other possibilities if the output isn't defined.
187 </para>
188
189 <variablelist>
190 <varlistentry>
191 <term>
192 <varname> $outputDev</varname>
193 </term>
194 <listitem>
195 <para>
196 is for development-only files. These include C(++) headers, pkg-config,
197 cmake and aclocal files. They go to <varname>dev</varname> or
198 <varname>out</varname> by default.
199 </para>
200 </listitem>
201 </varlistentry>
202 <varlistentry>
203 <term>
204 <varname> $outputBin</varname>
205 </term>
206 <listitem>
207 <para>
208 is meant for user-facing binaries, typically residing in bin/. They go
209 to <varname>bin</varname> or <varname>out</varname> by default.
210 </para>
211 </listitem>
212 </varlistentry>
213 <varlistentry>
214 <term>
215 <varname> $outputLib</varname>
216 </term>
217 <listitem>
218 <para>
219 is meant for libraries, typically residing in <filename>lib/</filename>
220 and <filename>libexec/</filename>. They go to <varname>lib</varname> or
221 <varname>out</varname> by default.
222 </para>
223 </listitem>
224 </varlistentry>
225 <varlistentry>
226 <term>
227 <varname> $outputDoc</varname>
228 </term>
229 <listitem>
230 <para>
231 is for user documentation, typically residing in
232 <filename>share/doc/</filename>. It goes to <varname>doc</varname> or
233 <varname>out</varname> by default.
234 </para>
235 </listitem>
236 </varlistentry>
237 <varlistentry>
238 <term>
239 <varname> $outputDevdoc</varname>
240 </term>
241 <listitem>
242 <para>
243 is for <emphasis>developer</emphasis> documentation. Currently we count
244 gtk-doc and devhelp books in there. It goes to <varname>devdoc</varname>
245 or is removed (!) by default. This is because e.g. gtk-doc tends to be
246 rather large and completely unused by nixpkgs users.
247 </para>
248 </listitem>
249 </varlistentry>
250 <varlistentry>
251 <term>
252 <varname> $outputMan</varname>
253 </term>
254 <listitem>
255 <para>
256 is for man pages (except for section 3). They go to
257 <varname>man</varname> or <varname>$outputBin</varname> by default.
258 </para>
259 </listitem>
260 </varlistentry>
261 <varlistentry>
262 <term>
263 <varname> $outputDevman</varname>
264 </term>
265 <listitem>
266 <para>
267 is for section 3 man pages. They go to <varname>devman</varname> or
268 <varname>$outputMan</varname> by default.
269 </para>
270 </listitem>
271 </varlistentry>
272 <varlistentry>
273 <term>
274 <varname> $outputInfo</varname>
275 </term>
276 <listitem>
277 <para>
278 is for info pages. They go to <varname>info</varname> or
279 <varname>$outputBin</varname> by default.
280 </para>
281 </listitem>
282 </varlistentry>
283 </variablelist>
284 </section>
285
286 <section xml:id="sec-multiple-outputs-caveats">
287 <title>Common caveats</title>
288
289 <itemizedlist>
290 <listitem>
291 <para>
292 Some configure scripts don't like some of the parameters passed by
293 default by the framework, e.g. <literal>--docdir=/foo/bar</literal>. You
294 can disable this by setting <literal>setOutputFlags = false;</literal>.
295 </para>
296 </listitem>
297 <listitem>
298 <para>
299 The outputs of a single derivation can retain references to each other,
300 but note that circular references are not allowed. (And each
301 strongly-connected component would act as a single output anyway.)
302 </para>
303 </listitem>
304 <listitem>
305 <para>
306 Most of split packages contain their core functionality in libraries.
307 These libraries tend to refer to various kind of data that typically gets
308 into <varname>out</varname>, e.g. locale strings, so there is often no
309 advantage in separating the libraries into <varname>lib</varname>, as
310 keeping them in <varname>out</varname> is easier.
311 </para>
312 </listitem>
313 <listitem>
314 <para>
315 Some packages have hidden assumptions on install paths, which complicates
316 splitting.
317 </para>
318 </listitem>
319 </itemizedlist>
320 </section>
321 </section>
322<!--Writing a split derivation-->
323</chapter>