pyrox.dev
1# BEAM Languages (Erlang, Elixir & LFE) {#sec-beam}
2
3## Introduction {#beam-introduction}
4
5In this document and related Nix expressions, we use the term, _BEAM_, to describe the environment. BEAM is the name of the Erlang Virtual Machine and, as far as we're concerned, from a packaging perspective, all languages that run on the BEAM are interchangeable. That which varies, like the build system, is transparent to users of any given BEAM package, so we make no distinction.
6
7## Available versions and deprecations schedule {#available-versions-and-deprecations-schedule}
8
9### Elixir {#elixir}
10
11nixpkgs follows the [official elixir deprecation schedule](https://hexdocs.pm/elixir/compatibility-and-deprecations.html) and keeps the last 5 released versions of Elixir available.
12
13## Structure {#beam-structure}
14
15All BEAM-related expressions are available via the top-level `beam` attribute, which includes:
16
17- `interpreters`: a set of compilers running on the BEAM, including multiple Erlang/OTP versions (`beam.interpreters.erlang_22`, etc), Elixir (`beam.interpreters.elixir`) and LFE (Lisp Flavoured Erlang) (`beam.interpreters.lfe`).
18
19- `packages`: a set of package builders (Mix and rebar3), each compiled with a specific Erlang/OTP version, e.g. `beam.packages.erlang22`.
20
21The default Erlang compiler, defined by `beam.interpreters.erlang`, is aliased as `erlang`. The default BEAM package set is defined by `beam.packages.erlang` and aliased at the top level as `beamPackages`.
22
23To create a package builder built with a custom Erlang version, use the lambda, `beam.packagesWith`, which accepts an Erlang/OTP derivation and produces a package builder similar to `beam.packages.erlang`.
24
25Many Erlang/OTP distributions available in `beam.interpreters` have versions with ODBC and/or Java enabled or without wx (no observer support). For example, there's `beam.interpreters.erlang_22_odbc_javac`, which corresponds to `beam.interpreters.erlang_22` and `beam.interpreters.erlang_22_nox`, which corresponds to `beam.interpreters.erlang_22`.
26
27## Build Tools {#build-tools}
28
29### Rebar3 {#build-tools-rebar3}
30
31We provide a version of Rebar3, under `rebar3`. We also provide a helper to fetch Rebar3 dependencies from a lockfile under `fetchRebar3Deps`.
32
33We also provide a version on Rebar3 with plugins included, under `rebar3WithPlugins`. This package is a function which takes two arguments: `plugins`, a list of nix derivations to include as plugins (loaded only when specified in `rebar.config`), and `globalPlugins`, which should always be loaded by rebar3. Example: `rebar3WithPlugins { globalPlugins = [beamPackages.pc]; }`.
34
35When adding a new plugin it is important that the `packageName` attribute is the same as the atom used by rebar3 to refer to the plugin.
36
37### Mix & Erlang.mk {#build-tools-other}
38
39Erlang.mk works exactly as expected. There is a bootstrap process that needs to be run, which is supported by the `buildErlangMk` derivation.
40
41For Elixir applications use `mixRelease` to make a release. See examples for more details.
42
43There is also a `buildMix` helper, whose behavior is closer to that of `buildErlangMk` and `buildRebar3`. The primary difference is that mixRelease makes a release, while buildMix only builds the package, making it useful for libraries and other dependencies.
44
45## How to Install BEAM Packages {#how-to-install-beam-packages}
46
47BEAM builders are not registered at the top level, because they are not relevant to the vast majority of Nix users.
48To use any of those builders into your environment, refer to them by their attribute path under `beamPackages`, e.g. `beamPackages.rebar3`:
49
50::: {.example #ex-beam-ephemeral-shell}
51# Ephemeral shell
52
53```ShellSession
54$ nix-shell -p beamPackages.rebar3
55```
56:::
57
58::: {.example #ex-beam-declarative-shell}
59# Declarative shell
60
61```nix
62let
63 pkgs = import <nixpkgs> { config = {}; overlays = []; };
64in
65pkgs.mkShell {
66 packages = [ pkgs.beamPackages.rebar3 ];
67}
68```
69:::
70
71## Packaging BEAM Applications {#packaging-beam-applications}
72
73### Erlang Applications {#packaging-erlang-applications}
74
75#### Rebar3 Packages {#rebar3-packages}
76
77The Nix function, `buildRebar3`, defined in `beam.packages.erlang.buildRebar3` and aliased at the top level, can be used to build a derivation that understands how to build a Rebar3 project.
78
79If a package needs to compile native code via Rebar3's port compilation mechanism, add `compilePort = true;` to the derivation.
80
81#### Erlang.mk Packages {#erlang-mk-packages}
82
83Erlang.mk functions similarly to Rebar3, except we use `buildErlangMk` instead of `buildRebar3`.
84
85#### Mix Packages {#mix-packages}
86
87`mixRelease` is used to make a release in the mix sense. Dependencies will need to be fetched with `fetchMixDeps` and passed to it.
88
89#### mixRelease - Elixir Phoenix example {#mix-release-elixir-phoenix-example}
90
91there are 3 steps, frontend dependencies (javascript), backend dependencies (elixir) and the final derivation that puts both of those together
92
93##### mixRelease - Frontend dependencies (javascript) {#mix-release-javascript-deps}
94
95For phoenix projects, inside of nixpkgs you can either use yarn2nix (mkYarnModule) or node2nix. An example with yarn2nix can be found [here](https://github.com/NixOS/nixpkgs/blob/master/pkgs/servers/web-apps/plausible/default.nix#L39). An example with node2nix will follow. To package something outside of nixpkgs, you have alternatives like [npmlock2nix](https://github.com/nix-community/npmlock2nix) or [nix-npm-buildpackage](https://github.com/serokell/nix-npm-buildpackage)
96
97##### mixRelease - backend dependencies (mix) {#mix-release-mix-deps}
98
99There are 2 ways to package backend dependencies. With mix2nix and with a fixed-output-derivation (FOD).
100
101###### mix2nix {#mix2nix}
102
103`mix2nix` is a cli tool available in nixpkgs. it will generate a nix expression from a mix.lock file. It is quite standard in the 2nix tool series.
104
105Note that currently mix2nix can't handle git dependencies inside the mix.lock file. If you have git dependencies, you can either add them manually (see [example](https://github.com/NixOS/nixpkgs/blob/master/pkgs/servers/pleroma/default.nix#L20)) or use the FOD method.
106
107The advantage of using mix2nix is that nix will know your whole dependency graph. On a dependency update, this won't trigger a full rebuild and download of all the dependencies, where FOD will do so.
108
109Practical steps:
110
111- run `mix2nix > mix_deps.nix` in the upstream repo.
112- pass `mixNixDeps = with pkgs; import ./mix_deps.nix { inherit lib beamPackages; };` as an argument to mixRelease.
113
114If there are git dependencies.
115
116- You'll need to fix the version artificially in mix.exs and regenerate the mix.lock with fixed version (on upstream). This will enable you to run `mix2nix > mix_deps.nix`.
117- From the mix_deps.nix file, remove the dependencies that had git versions and pass them as an override to the import function.
118
119```nix
120{
121 mixNixDeps = import ./mix.nix {
122 inherit beamPackages lib;
123 overrides = (final: prev: {
124 # mix2nix does not support git dependencies yet,
125 # so we need to add them manually
126 prometheus_ex = beamPackages.buildMix rec {
127 name = "prometheus_ex";
128 version = "3.0.5";
129
130 # Change the argument src with the git src that you actually need
131 src = fetchFromGitLab {
132 domain = "git.pleroma.social";
133 group = "pleroma";
134 owner = "elixir-libraries";
135 repo = "prometheus.ex";
136 rev = "a4e9beb3c1c479d14b352fd9d6dd7b1f6d7deee5";
137 hash = "sha256-U17LlN6aGUKUFnT4XyYXppRN+TvUBIBRHEUsfeIiGOw=";
138 };
139 # you can re-use the same beamDeps argument as generated
140 beamDeps = with final; [ prometheus ];
141 };
142 });
143 };
144}
145```
146
147You will need to run the build process once to fix the hash to correspond to your new git src.
148
149###### FOD {#fixed-output-derivation}
150
151A fixed output derivation will download mix dependencies from the internet. To ensure reproducibility, a hash will be supplied. Note that mix is relatively reproducible. An FOD generating a different hash on each run hasn't been observed (as opposed to npm where the chances are relatively high). See [elixir-ls](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/beam-modules/elixir-ls/default.nix) for a usage example of FOD.
152
153Practical steps
154
155- start with the following argument to mixRelease
156
157```nix
158{
159 mixFodDeps = fetchMixDeps {
160 pname = "mix-deps-${pname}";
161 inherit src version;
162 hash = lib.fakeHash;
163 };
164}
165```
166
167The first build will complain about the hash value, you can replace with the suggested value after that.
168
169Note that if after you've replaced the value, nix suggests another hash, then mix is not fetching the dependencies reproducibly. An FOD will not work in that case and you will have to use mix2nix.
170
171##### mixRelease - example {#mix-release-example}
172
173Here is how your `default.nix` file would look for a phoenix project.
174
175```nix
176with import <nixpkgs> { };
177
178let
179 # beam.interpreters.erlang_26 is available if you need a particular version
180 packages = beam.packagesWith beam.interpreters.erlang;
181
182 pname = "your_project";
183 version = "0.0.1";
184
185 src = builtins.fetchgit {
186 url = "ssh://git@github.com/your_id/your_repo";
187 rev = "replace_with_your_commit";
188 };
189
190 # if using mix2nix you can use the mixNixDeps attribute
191 mixFodDeps = packages.fetchMixDeps {
192 pname = "mix-deps-${pname}";
193 inherit src version;
194 # nix will complain and tell you the right value to replace this with
195 hash = lib.fakeHash;
196 mixEnv = ""; # default is "prod", when empty includes all dependencies, such as "dev", "test".
197 # if you have build time environment variables add them here
198 MY_ENV_VAR="my_value";
199 };
200
201 nodeDependencies = (pkgs.callPackage ./assets/default.nix { }).shell.nodeDependencies;
202
203in packages.mixRelease {
204 inherit src pname version mixFodDeps;
205 # if you have build time environment variables add them here
206 MY_ENV_VAR="my_value";
207
208 postBuild = ''
209 ln -sf ${nodeDependencies}/lib/node_modules assets/node_modules
210 npm run deploy --prefix ./assets
211
212 # for external task you need a workaround for the no deps check flag
213 # https://github.com/phoenixframework/phoenix/issues/2690
214 mix do deps.loadpaths --no-deps-check, phx.digest
215 mix phx.digest --no-deps-check
216 '';
217}
218```
219
220Setup will require the following steps:
221
222- Move your secrets to runtime environment variables. For more information refer to the [runtime.exs docs](https://hexdocs.pm/mix/Mix.Tasks.Release.html#module-runtime-configuration). On a fresh Phoenix build that would mean that both `DATABASE_URL` and `SECRET_KEY` need to be moved to `runtime.exs`.
223- `cd assets` and `nix-shell -p node2nix --run "node2nix --development"` will generate a Nix expression containing your frontend dependencies
224- commit and push those changes
225- you can now `nix-build .`
226- To run the release, set the `RELEASE_TMP` environment variable to a directory that your program has write access to. It will be used to store the BEAM settings.
227
228#### Example of creating a service for an Elixir - Phoenix project {#example-of-creating-a-service-for-an-elixir---phoenix-project}
229
230In order to create a service with your release, you could add a `service.nix`
231in your project with the following
232
233```nix
234{config, pkgs, lib, ...}:
235
236let
237 release = pkgs.callPackage ./default.nix;
238 release_name = "app";
239 working_directory = "/home/app";
240in
241{
242 systemd.services.${release_name} = {
243 wantedBy = [ "multi-user.target" ];
244 after = [ "network.target" "postgresql.service" ];
245 # note that if you are connecting to a postgres instance on a different host
246 # postgresql.service should not be included in the requires.
247 requires = [ "network-online.target" "postgresql.service" ];
248 description = "my app";
249 environment = {
250 # RELEASE_TMP is used to write the state of the
251 # VM configuration when the system is running
252 # it needs to be a writable directory
253 RELEASE_TMP = working_directory;
254 # can be generated in an elixir console with
255 # Base.encode32(:crypto.strong_rand_bytes(32))
256 RELEASE_COOKIE = "my_cookie";
257 MY_VAR = "my_var";
258 };
259 serviceConfig = {
260 Type = "exec";
261 DynamicUser = true;
262 WorkingDirectory = working_directory;
263 # Implied by DynamicUser, but just to emphasize due to RELEASE_TMP
264 PrivateTmp = true;
265 ExecStart = ''
266 ${release}/bin/${release_name} start
267 '';
268 ExecStop = ''
269 ${release}/bin/${release_name} stop
270 '';
271 ExecReload = ''
272 ${release}/bin/${release_name} restart
273 '';
274 Restart = "on-failure";
275 RestartSec = 5;
276 StartLimitBurst = 3;
277 StartLimitInterval = 10;
278 };
279 # disksup requires bash
280 path = [ pkgs.bash ];
281 };
282
283 # in case you have migration scripts or you want to use a remote shell
284 environment.systemPackages = [ release ];
285}
286```
287
288## How to Develop {#how-to-develop}
289
290### Creating a Shell {#creating-a-shell}
291
292Usually, we need to create a `shell.nix` file and do our development inside of the environment specified therein. Just install your version of Erlang and any other interpreters, and then use your normal build tools. As an example with Elixir:
293
294```nix
295{ pkgs ? import <nixpkgs> {} }:
296
297with pkgs;
298let
299 elixir = beam.packages.erlang_24.elixir_1_12;
300in
301mkShell {
302 buildInputs = [ elixir ];
303}
304```
305
306### Using an overlay {#beam-using-overlays}
307
308If you need to use an overlay to change some attributes of a derivation, e.g. if you need a bugfix from a version that is not yet available in nixpkgs, you can override attributes such as `version` (and the corresponding `hash`) and then use this overlay in your development environment:
309
310#### `shell.nix` {#beam-using-overlays-shell.nix}
311
312```nix
313let
314 elixir_1_13_1_overlay = (self: super: {
315 elixir_1_13 = super.elixir_1_13.override {
316 version = "1.13.1";
317 sha256 = "sha256-t0ic1LcC7EV3avWGdR7VbyX7pGDpnJSW1ZvwvQUPC3w=";
318 };
319 });
320 pkgs = import <nixpkgs> { overlays = [ elixir_1_13_1_overlay ]; };
321in
322with pkgs;
323mkShell {
324 buildInputs = [
325 elixir_1_13
326 ];
327}
328```
329
330#### Elixir - Phoenix project {#elixir---phoenix-project}
331
332Here is an example `shell.nix`.
333
334```nix
335with import <nixpkgs> { };
336
337let
338 # define packages to install
339 basePackages = [
340 git
341 # replace with beam.packages.erlang.elixir_1_13 if you need
342 beam.packages.erlang.elixir
343 nodejs
344 postgresql_14
345 # only used for frontend dependencies
346 # you are free to use yarn2nix as well
347 nodePackages.node2nix
348 # formatting js file
349 nodePackages.prettier
350 ];
351
352 inputs = basePackages ++ lib.optionals stdenv.isLinux [ inotify-tools ]
353 ++ lib.optionals stdenv.isDarwin
354 (with darwin.apple_sdk.frameworks; [ CoreFoundation CoreServices ]);
355
356 # define shell startup command
357 hooks = ''
358 # this allows mix to work on the local directory
359 mkdir -p .nix-mix .nix-hex
360 export MIX_HOME=$PWD/.nix-mix
361 export HEX_HOME=$PWD/.nix-mix
362 # make hex from Nixpkgs available
363 # `mix local.hex` will install hex into MIX_HOME and should take precedence
364 export MIX_PATH="${beam.packages.erlang.hex}/lib/erlang/lib/hex/ebin"
365 export PATH=$MIX_HOME/bin:$HEX_HOME/bin:$PATH
366 export LANG=C.UTF-8
367 # keep your shell history in iex
368 export ERL_AFLAGS="-kernel shell_history enabled"
369
370 # postges related
371 # keep all your db data in a folder inside the project
372 export PGDATA="$PWD/db"
373
374 # phoenix related env vars
375 export POOL_SIZE=15
376 export DB_URL="postgresql://postgres:postgres@localhost:5432/db"
377 export PORT=4000
378 export MIX_ENV=dev
379 # add your project env vars here, word readable in the nix store.
380 export ENV_VAR="your_env_var"
381 '';
382
383in mkShell {
384 buildInputs = inputs;
385 shellHook = hooks;
386}
387```
388
389Initializing the project will require the following steps:
390
391- create the db directory `initdb ./db` (inside your mix project folder)
392- create the postgres user `createuser postgres -ds`
393- create the db `createdb db`
394- start the postgres instance `pg_ctl -l "$PGDATA/server.log" start`
395- add the `/db` folder to your `.gitignore`
396- you can start your phoenix server and get a shell with `iex -S mix phx.server`