pyrox.dev
1# BEAM Languages (Erlang, Elixir & LFE) {#sec-beam}
2
3## Introduction {#beam-introduction}
4
5In this document and related Nix expressions, we use the term, _BEAM_, to describe the environment. BEAM is the name of the Erlang Virtual Machine and, as far as we're concerned, from a packaging perspective, all languages that run on the BEAM are interchangeable. That which varies, like the build system, is transparent to users of any given BEAM package, so we make no distinction.
6
7## Available versions and deprecations schedule {#available-versions-and-deprecations-schedule}
8
9### Elixir {#elixir}
10
11Nixpkgs follows the [official elixir deprecation schedule](https://hexdocs.pm/elixir/compatibility-and-deprecations.html) and keeps the last 5 released versions of Elixir available.
12
13## Structure {#beam-structure}
14
15All BEAM-related expressions are available via the top-level `beam` attribute, which includes:
16
17- `interpreters`: a set of compilers running on the BEAM, including multiple Erlang/OTP versions (`beam.interpreters.erlang_22`, etc), Elixir (`beam.interpreters.elixir`) and LFE (Lisp Flavoured Erlang) (`beam.interpreters.lfe`).
18
19- `packages`: a set of package builders (Mix and rebar3), each compiled with a specific Erlang/OTP version, e.g. `beam.packages.erlang22`.
20
21The default Erlang compiler, defined by `beam.interpreters.erlang`, is aliased as `erlang`. The default BEAM package set is defined by `beam.packages.erlang` and aliased at the top level as `beamPackages`.
22
23To create a package builder built with a custom Erlang version, use the lambda, `beam.packagesWith`, which accepts an Erlang/OTP derivation and produces a package builder similar to `beam.packages.erlang`.
24
25Many Erlang/OTP distributions available in `beam.interpreters` have versions with ODBC and/or Java enabled or without wx (no observer support). For example, there's `beam.interpreters.erlang_22_odbc_javac`, which corresponds to `beam.interpreters.erlang_22` and `beam.interpreters.erlang_22_nox`, which corresponds to `beam.interpreters.erlang_22`.
26
27## Build Tools {#build-tools}
28
29### Rebar3 {#build-tools-rebar3}
30
31We provide a version of Rebar3, under `rebar3`. We also provide a helper to fetch Rebar3 dependencies from a lockfile under `fetchRebar3Deps`.
32
33We also provide a version on Rebar3 with plugins included, under `rebar3WithPlugins`. This package is a function which takes two arguments: `plugins`, a list of nix derivations to include as plugins (loaded only when specified in `rebar.config`), and `globalPlugins`, which should always be loaded by rebar3. Example: `rebar3WithPlugins { globalPlugins = [beamPackages.pc]; }`.
34
35When adding a new plugin it is important that the `packageName` attribute is the same as the atom used by rebar3 to refer to the plugin.
36
37### Mix & Erlang.mk {#build-tools-other}
38
39Erlang.mk works exactly as expected. There is a bootstrap process that needs to be run, which is supported by the `buildErlangMk` derivation.
40
41For Elixir applications use `mixRelease` to make a release. See examples for more details.
42
43There is also a `buildMix` helper, whose behavior is closer to that of `buildErlangMk` and `buildRebar3`. The primary difference is that mixRelease makes a release, while buildMix only builds the package, making it useful for libraries and other dependencies.
44
45## How to Install BEAM Packages {#how-to-install-beam-packages}
46
47BEAM builders are not registered at the top level, because they are not relevant to the vast majority of Nix users.
48To use any of those builders into your environment, refer to them by their attribute path under `beamPackages`, e.g. `beamPackages.rebar3`:
49
50::: {.example #ex-beam-ephemeral-shell}
51# Ephemeral shell
52
53```ShellSession
54$ nix-shell -p beamPackages.rebar3
55```
56:::
57
58::: {.example #ex-beam-declarative-shell}
59# Declarative shell
60
61```nix
62let
63 pkgs = import <nixpkgs> {
64 config = { };
65 overlays = [ ];
66 };
67in
68pkgs.mkShell { packages = [ pkgs.beamPackages.rebar3 ]; }
69```
70:::
71
72## Packaging BEAM Applications {#packaging-beam-applications}
73
74### Erlang Applications {#packaging-erlang-applications}
75
76#### Rebar3 Packages {#rebar3-packages}
77
78The Nix function, `buildRebar3`, defined in `beam.packages.erlang.buildRebar3` and aliased at the top level, can be used to build a derivation that understands how to build a Rebar3 project.
79
80If a package needs to compile native code via Rebar3's port compilation mechanism, add `compilePort = true;` to the derivation.
81
82#### Erlang.mk Packages {#erlang-mk-packages}
83
84Erlang.mk functions similarly to Rebar3, except we use `buildErlangMk` instead of `buildRebar3`.
85
86#### Mix Packages {#mix-packages}
87
88`mixRelease` is used to make a release in the mix sense. Dependencies will need to be fetched with `fetchMixDeps` and passed to it.
89
90#### mixRelease - Elixir Phoenix example {#mix-release-elixir-phoenix-example}
91
92there are 3 steps: frontend dependencies (javascript), backend dependencies (elixir), and the final derivation that puts both of those together
93
94##### mixRelease - Frontend dependencies (javascript) {#mix-release-javascript-deps}
95
96For phoenix projects, inside of Nixpkgs you can either use yarn2nix (mkYarnModule) or node2nix. An example with yarn2nix can be found [here](https://github.com/NixOS/nixpkgs/blob/master/pkgs/servers/web-apps/plausible/default.nix#L39). An example with node2nix will follow. To package something outside of nixpkgs, you have alternatives like [npmlock2nix](https://github.com/nix-community/npmlock2nix) or [nix-npm-buildpackage](https://github.com/serokell/nix-npm-buildpackage)
97
98##### mixRelease - backend dependencies (mix) {#mix-release-mix-deps}
99
100There are 2 ways to package backend dependencies. With mix2nix and with a fixed-output-derivation (FOD).
101
102###### mix2nix {#mix2nix}
103
104`mix2nix` is a cli tool available in Nixpkgs. It will generate a Nix expression from a `mix.lock` file. It is quite standard in the 2nix tool series.
105
106Note that currently mix2nix can't handle git dependencies inside the mix.lock file. If you have git dependencies, you can either add them manually (see [example](https://github.com/NixOS/nixpkgs/blob/master/pkgs/servers/pleroma/default.nix#L20)) or use the FOD method.
107
108The advantage of using mix2nix is that nix will know your whole dependency graph. On a dependency update, this won't trigger a full rebuild and download of all the dependencies, where FOD will do so.
109
110Practical steps:
111
112- run `mix2nix > mix_deps.nix` in the upstream repo.
113- pass `mixNixDeps = with pkgs; import ./mix_deps.nix { inherit lib beamPackages; };` as an argument to mixRelease.
114
115If there are git dependencies.
116
117- You'll need to fix the version artificially in mix.exs and regenerate the mix.lock with fixed version (on upstream). This will enable you to run `mix2nix > mix_deps.nix`.
118- From the mix_deps.nix file, remove the dependencies that had git versions and pass them as an override to the import function.
119
120```nix
121{
122 mixNixDeps = import ./mix.nix {
123 inherit beamPackages lib;
124 overrides = (
125 final: prev: {
126 # mix2nix does not support git dependencies yet,
127 # so we need to add them manually
128 prometheus_ex = beamPackages.buildMix rec {
129 name = "prometheus_ex";
130 version = "3.0.5";
131
132 # Change the argument src with the git src that you actually need
133 src = fetchFromGitLab {
134 domain = "git.pleroma.social";
135 group = "pleroma";
136 owner = "elixir-libraries";
137 repo = "prometheus.ex";
138 rev = "a4e9beb3c1c479d14b352fd9d6dd7b1f6d7deee5";
139 hash = "sha256-U17LlN6aGUKUFnT4XyYXppRN+TvUBIBRHEUsfeIiGOw=";
140 };
141 # you can re-use the same beamDeps argument as generated
142 beamDeps = with final; [ prometheus ];
143 };
144 }
145 );
146 };
147}
148```
149
150You will need to run the build process once to fix the hash to correspond to your new git src.
151
152###### FOD {#fixed-output-derivation}
153
154A fixed output derivation will download mix dependencies from the internet. To ensure reproducibility, a hash will be supplied. Note that mix is relatively reproducible. An FOD generating a different hash on each run hasn't been observed (as opposed to npm where the chances are relatively high). See [elixir-ls](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/beam-modules/elixir-ls/default.nix) for a usage example of FOD.
155
156Practical steps
157
158- start with the following argument to mixRelease
159
160```nix
161{
162 mixFodDeps = fetchMixDeps {
163 pname = "mix-deps-${pname}";
164 inherit src version;
165 hash = lib.fakeHash;
166 };
167}
168```
169
170The first build will complain about the hash value, you can replace with the suggested value after that.
171
172Note that if after you've replaced the value, nix suggests another hash, then mix is not fetching the dependencies reproducibly. An FOD will not work in that case and you will have to use mix2nix.
173
174##### mixRelease - example {#mix-release-example}
175
176Here is how your `default.nix` file would look for a Phoenix project.
177
178```nix
179with import <nixpkgs> { };
180
181let
182 # beam.interpreters.erlang_26 is available if you need a particular version
183 packages = beam.packagesWith beam.interpreters.erlang;
184
185 pname = "your_project";
186 version = "0.0.1";
187
188 src = builtins.fetchgit {
189 url = "ssh://git@github.com/your_id/your_repo";
190 rev = "replace_with_your_commit";
191 };
192
193 # if using mix2nix you can use the mixNixDeps attribute
194 mixFodDeps = packages.fetchMixDeps {
195 pname = "mix-deps-${pname}";
196 inherit src version;
197 # nix will complain and tell you the right value to replace this with
198 hash = lib.fakeHash;
199 mixEnv = ""; # default is "prod", when empty includes all dependencies, such as "dev", "test".
200 # if you have build time environment variables add them here
201 MY_ENV_VAR = "my_value";
202 };
203
204 nodeDependencies = (pkgs.callPackage ./assets/default.nix { }).shell.nodeDependencies;
205
206in
207packages.mixRelease {
208 inherit
209 src
210 pname
211 version
212 mixFodDeps
213 ;
214 # if you have build time environment variables add them here
215 MY_ENV_VAR = "my_value";
216
217 postBuild = ''
218 ln -sf ${nodeDependencies}/lib/node_modules assets/node_modules
219 npm run deploy --prefix ./assets
220
221 # for external task you need a workaround for the no deps check flag
222 # https://github.com/phoenixframework/phoenix/issues/2690
223 mix do deps.loadpaths --no-deps-check, phx.digest
224 mix phx.digest --no-deps-check
225 '';
226}
227```
228
229Setup will require the following steps:
230
231- Move your secrets to runtime environment variables. For more information refer to the [runtime.exs docs](https://hexdocs.pm/mix/Mix.Tasks.Release.html#module-runtime-configuration). On a fresh Phoenix build that would mean that both `DATABASE_URL` and `SECRET_KEY` need to be moved to `runtime.exs`.
232- `cd assets` and `nix-shell -p node2nix --run "node2nix --development"` will generate a Nix expression containing your frontend dependencies
233- commit and push those changes
234- you can now `nix-build .`
235- To run the release, set the `RELEASE_TMP` environment variable to a directory that your program has write access to. It will be used to store the BEAM settings.
236
237#### Example of creating a service for an Elixir - Phoenix project {#example-of-creating-a-service-for-an-elixir---phoenix-project}
238
239In order to create a service with your release, you could add a `service.nix`
240in your project with the following
241
242```nix
243{
244 config,
245 pkgs,
246 lib,
247 ...
248}:
249
250let
251 release = pkgs.callPackage ./default.nix;
252 release_name = "app";
253 working_directory = "/home/app";
254in
255{
256 systemd.services.${release_name} = {
257 wantedBy = [ "multi-user.target" ];
258 after = [
259 "network.target"
260 "postgresql.target"
261 ];
262 # note that if you are connecting to a postgres instance on a different host
263 # postgresql.target should not be included in the requires.
264 requires = [
265 "network-online.target"
266 "postgresql.target"
267 ];
268 description = "my app";
269 environment = {
270 # RELEASE_TMP is used to write the state of the
271 # VM configuration when the system is running
272 # it needs to be a writable directory
273 RELEASE_TMP = working_directory;
274 # can be generated in an elixir console with
275 # Base.encode32(:crypto.strong_rand_bytes(32))
276 RELEASE_COOKIE = "my_cookie";
277 MY_VAR = "my_var";
278 };
279 serviceConfig = {
280 Type = "exec";
281 DynamicUser = true;
282 WorkingDirectory = working_directory;
283 # Implied by DynamicUser, but just to emphasize due to RELEASE_TMP
284 PrivateTmp = true;
285 ExecStart = ''
286 ${release}/bin/${release_name} start
287 '';
288 ExecStop = ''
289 ${release}/bin/${release_name} stop
290 '';
291 ExecReload = ''
292 ${release}/bin/${release_name} restart
293 '';
294 Restart = "on-failure";
295 RestartSec = 5;
296 };
297 unitConfig = {
298 StartLimitBurst = 3;
299 StartLimitInterval = 10;
300 };
301 # disksup requires bash
302 path = [ pkgs.bash ];
303 };
304
305 # in case you have migration scripts or you want to use a remote shell
306 environment.systemPackages = [ release ];
307}
308```
309
310## How to Develop {#how-to-develop}
311
312### Creating a Shell {#creating-a-shell}
313
314Usually, we need to create a `shell.nix` file and do our development inside the environment specified therein. Just install your version of Erlang and any other interpreters, and then use your normal build tools. As an example, with Elixir:
315
316```nix
317{
318 pkgs ? import <nixpkgs> { },
319}:
320
321with pkgs;
322let
323 elixir = beam.packages.erlang_27.elixir_1_18;
324in
325mkShell { buildInputs = [ elixir ]; }
326```
327
328### Using an overlay {#beam-using-overlays}
329
330If you need to use an overlay to change some attributes of a derivation, e.g. if you need a bugfix from a version that is not yet available in Nixpkgs, you can override attributes such as `version` (and the corresponding `hash`) and then use this overlay in your development environment:
331
332#### `shell.nix` {#beam-using-overlays-shell.nix}
333
334```nix
335let
336 elixir_1_18_1_overlay = (
337 self: super: {
338 elixir_1_18 = super.elixir_1_18.override {
339 version = "1.18.1";
340 sha256 = "sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=";
341 };
342 }
343 );
344 pkgs = import <nixpkgs> { overlays = [ elixir_1_18_1_overlay ]; };
345in
346with pkgs;
347mkShell { buildInputs = [ elixir_1_18 ]; }
348```
349
350#### Elixir - Phoenix project {#elixir---phoenix-project}
351
352Here is an example `shell.nix`.
353
354```nix
355with import <nixpkgs> { };
356
357let
358 # define packages to install
359 basePackages = [
360 git
361 # replace with beam.packages.erlang.elixir_1_18 if you need
362 beam.packages.erlang.elixir
363 nodejs
364 postgresql_14
365 # only used for frontend dependencies
366 # you are free to use yarn2nix as well
367 nodePackages.node2nix
368 # formatting js file
369 nodePackages.prettier
370 ];
371
372 inputs = basePackages ++ lib.optionals stdenv.hostPlatform.isLinux [ inotify-tools ];
373
374 # define shell startup command
375 hooks = ''
376 # this allows mix to work on the local directory
377 mkdir -p .nix-mix .nix-hex
378 export MIX_HOME=$PWD/.nix-mix
379 export HEX_HOME=$PWD/.nix-mix
380 # make hex from Nixpkgs available
381 # `mix local.hex` will install hex into MIX_HOME and should take precedence
382 export MIX_PATH="${beam.packages.erlang.hex}/lib/erlang/lib/hex/ebin"
383 export PATH=$MIX_HOME/bin:$HEX_HOME/bin:$PATH
384 export LANG=C.UTF-8
385 # keep your shell history in iex
386 export ERL_AFLAGS="-kernel shell_history enabled"
387
388 # postges related
389 # keep all your db data in a folder inside the project
390 export PGDATA="$PWD/db"
391
392 # phoenix related env vars
393 export POOL_SIZE=15
394 export DB_URL="postgresql://postgres:postgres@localhost:5432/db"
395 export PORT=4000
396 export MIX_ENV=dev
397 # add your project env vars here, word readable in the nix store.
398 export ENV_VAR="your_env_var"
399 '';
400
401in
402mkShell {
403 buildInputs = inputs;
404 shellHook = hooks;
405}
406```
407
408Initializing the project will require the following steps:
409
410- create the db directory `initdb ./db` (inside your mix project folder)
411- create the postgres user `createuser postgres -ds`
412- create the db `createdb db`
413- start the postgres instance `pg_ctl -l "$PGDATA/server.log" start`
414- add the `/db` folder to your `.gitignore`
415- you can start your Phoenix server and get a shell with `iex -S mix phx.server`