pyrox.dev
1<section xmlns="http://docbook.org/ns/docbook"
2 xmlns:xlink="http://www.w3.org/1999/xlink"
3 xml:id="sec-beam">
4 <title>BEAM Languages (Erlang, Elixir & LFE)</title>
5
6 <section xml:id="beam-introduction">
7 <title>Introduction</title>
8
9 <para>
10 In this document and related Nix expressions, we use the term,
11 <emphasis>BEAM</emphasis>, to describe the environment. BEAM is the name of
12 the Erlang Virtual Machine and, as far as we're concerned, from a packaging
13 perspective, all languages that run on the BEAM are interchangeable. That
14 which varies, like the build system, is transparent to users of any given
15 BEAM package, so we make no distinction.
16 </para>
17 </section>
18
19 <section xml:id="beam-structure">
20 <title>Structure</title>
21
22 <para>
23 All BEAM-related expressions are available via the top-level
24 <literal>beam</literal> attribute, which includes:
25 </para>
26
27 <itemizedlist>
28 <listitem>
29 <para>
30 <literal>interpreters</literal>: a set of compilers running on the BEAM,
31 including multiple Erlang/OTP versions
32 (<literal>beam.interpreters.erlangR19</literal>, etc), Elixir
33 (<literal>beam.interpreters.elixir</literal>) and LFE
34 (<literal>beam.interpreters.lfe</literal>).
35 </para>
36 </listitem>
37 <listitem>
38 <para>
39 <literal>packages</literal>: a set of package sets, each compiled with a
40 specific Erlang/OTP version, e.g.
41 <literal>beam.packages.erlangR19</literal>.
42 </para>
43 </listitem>
44 </itemizedlist>
45
46 <para>
47 The default Erlang compiler, defined by
48 <literal>beam.interpreters.erlang</literal>, is aliased as
49 <literal>erlang</literal>. The default BEAM package set is defined by
50 <literal>beam.packages.erlang</literal> and aliased at the top level as
51 <literal>beamPackages</literal>.
52 </para>
53
54 <para>
55 To create a package set built with a custom Erlang version, use the lambda,
56 <literal>beam.packagesWith</literal>, which accepts an Erlang/OTP derivation
57 and produces a package set similar to
58 <literal>beam.packages.erlang</literal>.
59 </para>
60
61 <para>
62 Many Erlang/OTP distributions available in
63 <literal>beam.interpreters</literal> have versions with ODBC and/or Java
64 enabled. For example, there's
65 <literal>beam.interpreters.erlangR19_odbc_javac</literal>, which corresponds
66 to <literal>beam.interpreters.erlangR19</literal>.
67 </para>
68
69 <para xml:id="erlang-call-package">
70 We also provide the lambda,
71 <literal>beam.packages.erlang.callPackage</literal>, which simplifies
72 writing BEAM package definitions by injecting all packages from
73 <literal>beam.packages.erlang</literal> into the top-level context.
74 </para>
75 </section>
76
77 <section xml:id="build-tools">
78 <title>Build Tools</title>
79
80 <section xml:id="build-tools-rebar3">
81 <title>Rebar3</title>
82
83 <para>
84 By default, Rebar3 wants to manage its own dependencies. This is perfectly
85 acceptable in the normal, non-Nix setup, but in the Nix world, it is not.
86 To rectify this, we provide two versions of Rebar3:
87 <itemizedlist>
88 <listitem>
89 <para>
90 <literal>rebar3</literal>: patched to remove the ability to download
91 anything. When not running it via <literal>nix-shell</literal> or
92 <literal>nix-build</literal>, it's probably not going to work as
93 desired.
94 </para>
95 </listitem>
96 <listitem>
97 <para>
98 <literal>rebar3-open</literal>: the normal, unmodified Rebar3. It should
99 work exactly as would any other version of Rebar3. Any Erlang package
100 should rely on <literal>rebar3</literal> instead. See
101 <xref
102 linkend="rebar3-packages"/>.
103 </para>
104 </listitem>
105 </itemizedlist>
106 </para>
107 </section>
108
109 <section xml:id="build-tools-other">
110 <title>Mix & Erlang.mk</title>
111
112 <para>
113 Both Mix and Erlang.mk work exactly as expected. There is a bootstrap
114 process that needs to be run for both, however, which is supported by the
115 <literal>buildMix</literal> and <literal>buildErlangMk</literal>
116 derivations, respectively.
117 </para>
118 </section>
119 </section>
120
121 <section xml:id="how-to-install-beam-packages">
122 <title>How to Install BEAM Packages</title>
123
124 <para>
125 BEAM packages are not registered at the top level, simply because they are
126 not relevant to the vast majority of Nix users. They are installable using
127 the <literal>beam.packages.erlang</literal> attribute set (aliased as
128 <literal>beamPackages</literal>), which points to packages built by the
129 default Erlang/OTP version in Nixpkgs, as defined by
130 <literal>beam.interpreters.erlang</literal>. To list the available packages
131 in <literal>beamPackages</literal>, use the following command:
132 </para>
133
134<programlisting>
135$ nix-env -f "<nixpkgs>" -qaP -A beamPackages
136beamPackages.esqlite esqlite-0.2.1
137beamPackages.goldrush goldrush-0.1.7
138beamPackages.ibrowse ibrowse-4.2.2
139beamPackages.jiffy jiffy-0.14.5
140beamPackages.lager lager-3.0.2
141beamPackages.meck meck-0.8.3
142beamPackages.rebar3-pc pc-1.1.0
143 </programlisting>
144
145 <para>
146 To install any of those packages into your profile, refer to them by their
147 attribute path (first column):
148 </para>
149
150<programlisting>
151$ nix-env -f "<nixpkgs>" -iA beamPackages.ibrowse
152 </programlisting>
153
154 <para>
155 The attribute path of any BEAM package corresponds to the name of that
156 particular package in <link xlink:href="https://hex.pm">Hex</link> or its
157 OTP Application/Release name.
158 </para>
159 </section>
160
161 <section xml:id="packaging-beam-applications">
162 <title>Packaging BEAM Applications</title>
163
164 <section xml:id="packaging-erlang-applications">
165 <title>Erlang Applications</title>
166
167 <section xml:id="rebar3-packages">
168 <title>Rebar3 Packages</title>
169
170 <para>
171 The Nix function, <literal>buildRebar3</literal>, defined in
172 <literal>beam.packages.erlang.buildRebar3</literal> and aliased at the top
173 level, can be used to build a derivation that understands how to build a
174 Rebar3 project. For example, we can build
175 <link
176 xlink:href="https://github.com/erlang-nix/hex2nix">hex2nix</link>
177 as follows:
178 </para>
179
180<programlisting>
181 { stdenv, fetchFromGitHub, buildRebar3, ibrowse, jsx, erlware_commons }:
182
183 buildRebar3 rec {
184 name = "hex2nix";
185 version = "0.0.1";
186
187 src = fetchFromGitHub {
188 owner = "ericbmerritt";
189 repo = "hex2nix";
190 rev = "${version}";
191 sha256 = "1w7xjidz1l5yjmhlplfx7kphmnpvqm67w99hd2m7kdixwdxq0zqg";
192 };
193
194 beamDeps = [ ibrowse jsx erlware_commons ];
195 }
196 </programlisting>
197
198 <para>
199 Such derivations are callable with
200 <literal>beam.packages.erlang.callPackage</literal> (see
201 <xref
202 linkend="erlang-call-package"/>). To call this package using
203 the normal <literal>callPackage</literal>, refer to dependency packages
204 via <literal>beamPackages</literal>, e.g.
205 <literal>beamPackages.ibrowse</literal>.
206 </para>
207
208 <para>
209 Notably, <literal>buildRebar3</literal> includes
210 <literal>beamDeps</literal>, while <literal>stdenv.mkDerivation</literal>
211 does not. BEAM dependencies added there will be correctly handled by the
212 system.
213 </para>
214
215 <para>
216 If a package needs to compile native code via Rebar3's port compilation
217 mechanism, add <literal>compilePort = true;</literal> to the derivation.
218 </para>
219 </section>
220
221 <section xml:id="erlang-mk-packages">
222 <title>Erlang.mk Packages</title>
223
224 <para>
225 Erlang.mk functions similarly to Rebar3, except we use
226 <literal>buildErlangMk</literal> instead of
227 <literal>buildRebar3</literal>.
228 </para>
229
230<programlisting>
231 { buildErlangMk, fetchHex, cowlib, ranch }:
232
233 buildErlangMk {
234 name = "cowboy";
235 version = "1.0.4";
236
237 src = fetchHex {
238 pkg = "cowboy";
239 version = "1.0.4";
240 sha256 = "6a0edee96885fae3a8dd0ac1f333538a42e807db638a9453064ccfdaa6b9fdac";
241 };
242
243 beamDeps = [ cowlib ranch ];
244
245 meta = {
246 description = ''
247 Small, fast, modular HTTP server written in Erlang
248 '';
249 license = stdenv.lib.licenses.isc;
250 homepage = https://github.com/ninenines/cowboy;
251 };
252 }
253 </programlisting>
254 </section>
255
256 <section xml:id="mix-packages">
257 <title>Mix Packages</title>
258
259 <para>
260 Mix functions similarly to Rebar3, except we use
261 <literal>buildMix</literal> instead of <literal>buildRebar3</literal>.
262 </para>
263
264<programlisting>
265 { buildMix, fetchHex, plug, absinthe }:
266
267 buildMix {
268 name = "absinthe_plug";
269 version = "1.0.0";
270
271 src = fetchHex {
272 pkg = "absinthe_plug";
273 version = "1.0.0";
274 sha256 = "08459823fe1fd4f0325a8bf0c937a4520583a5a26d73b193040ab30a1dfc0b33";
275 };
276
277 beamDeps = [ plug absinthe ];
278
279 meta = {
280 description = ''
281 A plug for Absinthe, an experimental GraphQL toolkit
282 '';
283 license = stdenv.lib.licenses.bsd3;
284 homepage = https://github.com/CargoSense/absinthe_plug;
285 };
286 }
287 </programlisting>
288
289 <para>
290 Alternatively, we can use <literal>buildHex</literal> as a shortcut:
291 </para>
292
293<programlisting>
294 { buildHex, buildMix, plug, absinthe }:
295
296 buildHex {
297 name = "absinthe_plug";
298 version = "1.0.0";
299
300 sha256 = "08459823fe1fd4f0325a8bf0c937a4520583a5a26d73b193040ab30a1dfc0b33";
301
302 builder = buildMix;
303
304 beamDeps = [ plug absinthe ];
305
306 meta = {
307 description = ''
308 A plug for Absinthe, an experimental GraphQL toolkit
309 '';
310 license = stdenv.lib.licenses.bsd3;
311 homepage = https://github.com/CargoSense/absinthe_plug;
312 };
313 }
314 </programlisting>
315 </section>
316 </section>
317 </section>
318
319 <section xml:id="how-to-develop">
320 <title>How to Develop</title>
321
322 <section xml:id="accessing-an-environment">
323 <title>Accessing an Environment</title>
324
325 <para>
326 Often, we simply want to access a valid environment that contains a
327 specific package and its dependencies. We can accomplish that with the
328 <literal>env</literal> attribute of a derivation. For example, let's say we
329 want to access an Erlang REPL with <literal>ibrowse</literal> loaded up. We
330 could do the following:
331 </para>
332
333<programlisting>
334 $ nix-shell -A beamPackages.ibrowse.env --run "erl"
335 Erlang/OTP 18 [erts-7.0] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false]
336
337 Eshell V7.0 (abort with ^G)
338 1> m(ibrowse).
339 Module: ibrowse
340 MD5: 3b3e0137d0cbb28070146978a3392945
341 Compiled: January 10 2016, 23:34
342 Object file: /nix/store/g1rlf65rdgjs4abbyj4grp37ry7ywivj-ibrowse-4.2.2/lib/erlang/lib/ibrowse-4.2.2/ebin/ibrowse.beam
343 Compiler options: [{outdir,"/tmp/nix-build-ibrowse-4.2.2.drv-0/hex-source-ibrowse-4.2.2/_build/default/lib/ibrowse/ebin"},
344 debug_info,debug_info,nowarn_shadow_vars,
345 warn_unused_import,warn_unused_vars,warnings_as_errors,
346 {i,"/tmp/nix-build-ibrowse-4.2.2.drv-0/hex-source-ibrowse-4.2.2/_build/default/lib/ibrowse/include"}]
347 Exports:
348 add_config/1 send_req_direct/7
349 all_trace_off/0 set_dest/3
350 code_change/3 set_max_attempts/3
351 get_config_value/1 set_max_pipeline_size/3
352 get_config_value/2 set_max_sessions/3
353 get_metrics/0 show_dest_status/0
354 get_metrics/2 show_dest_status/1
355 handle_call/3 show_dest_status/2
356 handle_cast/2 spawn_link_worker_process/1
357 handle_info/2 spawn_link_worker_process/2
358 init/1 spawn_worker_process/1
359 module_info/0 spawn_worker_process/2
360 module_info/1 start/0
361 rescan_config/0 start_link/0
362 rescan_config/1 stop/0
363 send_req/3 stop_worker_process/1
364 send_req/4 stream_close/1
365 send_req/5 stream_next/1
366 send_req/6 terminate/2
367 send_req_direct/4 trace_off/0
368 send_req_direct/5 trace_off/2
369 send_req_direct/6 trace_on/0
370 trace_on/2
371 ok
372 2>
373 </programlisting>
374
375 <para>
376 Notice the <literal>-A beamPackages.ibrowse.env</literal>. That is the key
377 to this functionality.
378 </para>
379 </section>
380
381 <section xml:id="creating-a-shell">
382 <title>Creating a Shell</title>
383
384 <para>
385 Getting access to an environment often isn't enough to do real development.
386 Usually, we need to create a <literal>shell.nix</literal> file and do our
387 development inside of the environment specified therein. This file looks a
388 lot like the packaging described above, except that <literal>src</literal>
389 points to the project root and we call the package directly.
390 </para>
391
392<programlisting>
393{ pkgs ? import "<nixpkgs"> {} }:
394
395with pkgs;
396
397let
398
399 f = { buildRebar3, ibrowse, jsx, erlware_commons }:
400 buildRebar3 {
401 name = "hex2nix";
402 version = "0.1.0";
403 src = ./.;
404 beamDeps = [ ibrowse jsx erlware_commons ];
405 };
406 drv = beamPackages.callPackage f {};
407
408in
409
410 drv
411 </programlisting>
412
413 <section xml:id="building-in-a-shell">
414 <title>Building in a Shell (for Mix Projects)</title>
415
416 <para>
417 We can leverage the support of the derivation, irrespective of the build
418 derivation, by calling the commands themselves.
419 </para>
420
421<programlisting>
422# =============================================================================
423# Variables
424# =============================================================================
425
426NIX_TEMPLATES := "$(CURDIR)/nix-templates"
427
428TARGET := "$(PREFIX)"
429
430PROJECT_NAME := thorndyke
431
432NIXPKGS=../nixpkgs
433NIX_PATH=nixpkgs=$(NIXPKGS)
434NIX_SHELL=nix-shell -I "$(NIX_PATH)" --pure
435# =============================================================================
436# Rules
437# =============================================================================
438.PHONY= all test clean repl shell build test analyze configure install \
439 test-nix-install publish plt analyze
440
441all: build
442
443guard-%:
444 @ if [ "${${*}}" == "" ]; then \
445 echo "Environment variable $* not set"; \
446 exit 1; \
447 fi
448
449clean:
450 rm -rf _build
451 rm -rf .cache
452
453repl:
454 $(NIX_SHELL) --run "iex -pa './_build/prod/lib/*/ebin'"
455
456shell:
457 $(NIX_SHELL)
458
459configure:
460 $(NIX_SHELL) --command 'eval "$$configurePhase"'
461
462build: configure
463 $(NIX_SHELL) --command 'eval "$$buildPhase"'
464
465install:
466 $(NIX_SHELL) --command 'eval "$$installPhase"'
467
468test:
469 $(NIX_SHELL) --command 'mix test --no-start --no-deps-check'
470
471plt:
472 $(NIX_SHELL) --run "mix dialyzer.plt --no-deps-check"
473
474analyze: build plt
475 $(NIX_SHELL) --run "mix dialyzer --no-compile"
476
477 </programlisting>
478
479 <para>
480 Using a <literal>shell.nix</literal> as described (see
481 <xref
482 linkend="creating-a-shell"/>) should just work. Aside from
483 <literal>test</literal>, <literal>plt</literal>, and
484 <literal>analyze</literal>, the Make targets work just fine for all of the
485 build derivations.
486 </para>
487 </section>
488 </section>
489 </section>
490
491 <section xml:id="generating-packages-from-hex-with-hex2nix">
492 <title>Generating Packages from Hex with <literal>hex2nix</literal></title>
493
494 <para>
495 Updating the <link xlink:href="https://hex.pm">Hex</link> package set
496 requires
497 <link
498 xlink:href="https://github.com/erlang-nix/hex2nix">hex2nix</link>.
499 Given the path to the Erlang modules (usually
500 <literal>pkgs/development/erlang-modules</literal>), it will dump a file
501 called <literal>hex-packages.nix</literal>, containing all the packages that
502 use a recognized build system in
503 <link
504 xlink:href="https://hex.pm">Hex</link>. It can't be determined,
505 however, whether every package is buildable.
506 </para>
507
508 <para>
509 To make life easier for our users, try to build every
510 <link
511 xlink:href="https://hex.pm">Hex</link> package and remove those
512 that fail. To do that, simply run the following command in the root of your
513 <literal>nixpkgs</literal> repository:
514 </para>
515
516<programlisting>
517$ nix-build -A beamPackages
518 </programlisting>
519
520 <para>
521 That will attempt to build every package in <literal>beamPackages</literal>.
522 Then manually remove those that fail. Hopefully, someone will improve
523 <link
524 xlink:href="https://github.com/erlang-nix/hex2nix">hex2nix</link>
525 in the future to automate the process.
526 </para>
527 </section>
528</section>