pyrox.dev
1# BEAM Languages (Erlang, Elixir & LFE) {#sec-beam}
2
3## Introduction {#beam-introduction}
4
5In this document and related Nix expressions, we use the term, _BEAM_, to describe the environment. BEAM is the name of the Erlang Virtual Machine and, as far as we're concerned, from a packaging perspective, all languages that run on the BEAM are interchangeable. That which varies, like the build system, is transparent to users of any given BEAM package, so we make no distinction.
6
7## Available versions and deprecations schedule {#available-versions-and-deprecations-schedule}
8
9### Elixir {#elixir}
10
11nixpkgs follows the [official elixir deprecation schedule](https://hexdocs.pm/elixir/compatibility-and-deprecations.html) and keeps the last 5 released versions of Elixir available.
12
13## Structure {#beam-structure}
14
15All BEAM-related expressions are available via the top-level `beam` attribute, which includes:
16
17- `interpreters`: a set of compilers running on the BEAM, including multiple Erlang/OTP versions (`beam.interpreters.erlang_22`, etc), Elixir (`beam.interpreters.elixir`) and LFE (Lisp Flavoured Erlang) (`beam.interpreters.lfe`).
18
19- `packages`: a set of package builders (Mix and rebar3), each compiled with a specific Erlang/OTP version, e.g. `beam.packages.erlang22`.
20
21The default Erlang compiler, defined by `beam.interpreters.erlang`, is aliased as `erlang`. The default BEAM package set is defined by `beam.packages.erlang` and aliased at the top level as `beamPackages`.
22
23To create a package builder built with a custom Erlang version, use the lambda, `beam.packagesWith`, which accepts an Erlang/OTP derivation and produces a package builder similar to `beam.packages.erlang`.
24
25Many Erlang/OTP distributions available in `beam.interpreters` have versions with ODBC and/or Java enabled or without wx (no observer support). For example, there's `beam.interpreters.erlang_22_odbc_javac`, which corresponds to `beam.interpreters.erlang_22` and `beam.interpreters.erlang_22_nox`, which corresponds to `beam.interpreters.erlang_22`.
26
27## Build Tools {#build-tools}
28
29### Rebar3 {#build-tools-rebar3}
30
31We provide a version of Rebar3, under `rebar3`. We also provide a helper to fetch Rebar3 dependencies from a lockfile under `fetchRebar3Deps`.
32
33We also provide a version on Rebar3 with plugins included, under `rebar3WithPlugins`. This package is a function which takes two arguments: `plugins`, a list of nix derivations to include as plugins (loaded only when specified in `rebar.config`), and `globalPlugins`, which should always be loaded by rebar3. Example: `rebar3WithPlugins { globalPlugins = [beamPackages.pc]; }`.
34
35When adding a new plugin it is important that the `packageName` attribute is the same as the atom used by rebar3 to refer to the plugin.
36
37### Mix & Erlang.mk {#build-tools-other}
38
39Erlang.mk works exactly as expected. There is a bootstrap process that needs to be run, which is supported by the `buildErlangMk` derivation.
40
41For Elixir applications use `mixRelease` to make a release. See examples for more details.
42
43There is also a `buildMix` helper, whose behavior is closer to that of `buildErlangMk` and `buildRebar3`. The primary difference is that mixRelease makes a release, while buildMix only builds the package, making it useful for libraries and other dependencies.
44
45## How to Install BEAM Packages {#how-to-install-beam-packages}
46
47BEAM builders are not registered at the top level, simply because they are not relevant to the vast majority of Nix users. To install any of those builders into your profile, refer to them by their attribute path `beamPackages.rebar3`:
48
49```ShellSession
50$ nix-env -f "<nixpkgs>" -iA beamPackages.rebar3
51```
52
53## Packaging BEAM Applications {#packaging-beam-applications}
54
55### Erlang Applications {#packaging-erlang-applications}
56
57#### Rebar3 Packages {#rebar3-packages}
58
59The Nix function, `buildRebar3`, defined in `beam.packages.erlang.buildRebar3` and aliased at the top level, can be used to build a derivation that understands how to build a Rebar3 project.
60
61If a package needs to compile native code via Rebar3's port compilation mechanism, add `compilePort = true;` to the derivation.
62
63#### Erlang.mk Packages {#erlang-mk-packages}
64
65Erlang.mk functions similarly to Rebar3, except we use `buildErlangMk` instead of `buildRebar3`.
66
67#### Mix Packages {#mix-packages}
68
69`mixRelease` is used to make a release in the mix sense. Dependencies will need to be fetched with `fetchMixDeps` and passed to it.
70
71#### mixRelease - Elixir Phoenix example {#mix-release-elixir-phoenix-example}
72
73there are 3 steps, frontend dependencies (javascript), backend dependencies (elixir) and the final derivation that puts both of those together
74
75##### mixRelease - Frontend dependencies (javascript) {#mix-release-javascript-deps}
76
77For phoenix projects, inside of nixpkgs you can either use yarn2nix (mkYarnModule) or node2nix. An example with yarn2nix can be found [here](https://github.com/NixOS/nixpkgs/blob/master/pkgs/servers/web-apps/plausible/default.nix#L39). An example with node2nix will follow. To package something outside of nixpkgs, you have alternatives like [npmlock2nix](https://github.com/nix-community/npmlock2nix) or [nix-npm-buildpackage](https://github.com/serokell/nix-npm-buildpackage)
78
79##### mixRelease - backend dependencies (mix) {#mix-release-mix-deps}
80
81There are 2 ways to package backend dependencies. With mix2nix and with a fixed-output-derivation (FOD).
82
83###### mix2nix {#mix2nix}
84
85`mix2nix` is a cli tool available in nixpkgs. it will generate a nix expression from a mix.lock file. It is quite standard in the 2nix tool series.
86
87Note that currently mix2nix can't handle git dependencies inside the mix.lock file. If you have git dependencies, you can either add them manually (see [example](https://github.com/NixOS/nixpkgs/blob/master/pkgs/servers/pleroma/default.nix#L20)) or use the FOD method.
88
89The advantage of using mix2nix is that nix will know your whole dependency graph. On a dependency update, this won't trigger a full rebuild and download of all the dependencies, where FOD will do so.
90
91Practical steps:
92
93- run `mix2nix > mix_deps.nix` in the upstream repo.
94- pass `mixNixDeps = with pkgs; import ./mix_deps.nix { inherit lib beamPackages; };` as an argument to mixRelease.
95
96If there are git dependencies.
97
98- You'll need to fix the version artificially in mix.exs and regenerate the mix.lock with fixed version (on upstream). This will enable you to run `mix2nix > mix_deps.nix`.
99- From the mix_deps.nix file, remove the dependencies that had git versions and pass them as an override to the import function.
100
101```nix
102 mixNixDeps = import ./mix.nix {
103 inherit beamPackages lib;
104 overrides = (final: prev: {
105 # mix2nix does not support git dependencies yet,
106 # so we need to add them manually
107 prometheus_ex = beamPackages.buildMix rec {
108 name = "prometheus_ex";
109 version = "3.0.5";
110
111 # Change the argument src with the git src that you actually need
112 src = fetchFromGitLab {
113 domain = "git.pleroma.social";
114 group = "pleroma";
115 owner = "elixir-libraries";
116 repo = "prometheus.ex";
117 rev = "a4e9beb3c1c479d14b352fd9d6dd7b1f6d7deee5";
118 hash = "sha256-U17LlN6aGUKUFnT4XyYXppRN+TvUBIBRHEUsfeIiGOw=";
119 };
120 # you can re-use the same beamDeps argument as generated
121 beamDeps = with final; [ prometheus ];
122 };
123 });
124};
125```
126
127You will need to run the build process once to fix the hash to correspond to your new git src.
128
129###### FOD {#fixed-output-derivation}
130
131A fixed output derivation will download mix dependencies from the internet. To ensure reproducibility, a hash will be supplied. Note that mix is relatively reproducible. An FOD generating a different hash on each run hasn't been observed (as opposed to npm where the chances are relatively high). See [elixir-ls](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/beam-modules/elixir-ls/default.nix) for a usage example of FOD.
132
133Practical steps
134
135- start with the following argument to mixRelease
136
137```nix
138 mixFodDeps = fetchMixDeps {
139 pname = "mix-deps-${pname}";
140 inherit src version;
141 hash = lib.fakeHash;
142 };
143```
144
145The first build will complain about the hash value, you can replace with the suggested value after that.
146
147Note that if after you've replaced the value, nix suggests another hash, then mix is not fetching the dependencies reproducibly. An FOD will not work in that case and you will have to use mix2nix.
148
149##### mixRelease - example {#mix-release-example}
150
151Here is how your `default.nix` file would look for a phoenix project.
152
153```nix
154with import <nixpkgs> { };
155
156let
157 # beam.interpreters.erlang_23 is available if you need a particular version
158 packages = beam.packagesWith beam.interpreters.erlang;
159
160 pname = "your_project";
161 version = "0.0.1";
162
163 src = builtins.fetchgit {
164 url = "ssh://git@github.com/your_id/your_repo";
165 rev = "replace_with_your_commit";
166 };
167
168 # if using mix2nix you can use the mixNixDeps attribute
169 mixFodDeps = packages.fetchMixDeps {
170 pname = "mix-deps-${pname}";
171 inherit src version;
172 # nix will complain and tell you the right value to replace this with
173 hash = lib.fakeHash;
174 mixEnv = ""; # default is "prod", when empty includes all dependencies, such as "dev", "test".
175 # if you have build time environment variables add them here
176 MY_ENV_VAR="my_value";
177 };
178
179 nodeDependencies = (pkgs.callPackage ./assets/default.nix { }).shell.nodeDependencies;
180
181in packages.mixRelease {
182 inherit src pname version mixFodDeps;
183 # if you have build time environment variables add them here
184 MY_ENV_VAR="my_value";
185
186 postBuild = ''
187 ln -sf ${nodeDependencies}/lib/node_modules assets/node_modules
188 npm run deploy --prefix ./assets
189
190 # for external task you need a workaround for the no deps check flag
191 # https://github.com/phoenixframework/phoenix/issues/2690
192 mix do deps.loadpaths --no-deps-check, phx.digest
193 mix phx.digest --no-deps-check
194 '';
195}
196```
197
198Setup will require the following steps:
199
200- Move your secrets to runtime environment variables. For more information refer to the [runtime.exs docs](https://hexdocs.pm/mix/Mix.Tasks.Release.html#module-runtime-configuration). On a fresh Phoenix build that would mean that both `DATABASE_URL` and `SECRET_KEY` need to be moved to `runtime.exs`.
201- `cd assets` and `nix-shell -p node2nix --run node2nix --development` will generate a Nix expression containing your frontend dependencies
202- commit and push those changes
203- you can now `nix-build .`
204- To run the release, set the `RELEASE_TMP` environment variable to a directory that your program has write access to. It will be used to store the BEAM settings.
205
206#### Example of creating a service for an Elixir - Phoenix project {#example-of-creating-a-service-for-an-elixir---phoenix-project}
207
208In order to create a service with your release, you could add a `service.nix`
209in your project with the following
210
211```nix
212{config, pkgs, lib, ...}:
213
214let
215 release = pkgs.callPackage ./default.nix;
216 release_name = "app";
217 working_directory = "/home/app";
218in
219{
220 systemd.services.${release_name} = {
221 wantedBy = [ "multi-user.target" ];
222 after = [ "network.target" "postgresql.service" ];
223 # note that if you are connecting to a postgres instance on a different host
224 # postgresql.service should not be included in the requires.
225 requires = [ "network-online.target" "postgresql.service" ];
226 description = "my app";
227 environment = {
228 # RELEASE_TMP is used to write the state of the
229 # VM configuration when the system is running
230 # it needs to be a writable directory
231 RELEASE_TMP = working_directory;
232 # can be generated in an elixir console with
233 # Base.encode32(:crypto.strong_rand_bytes(32))
234 RELEASE_COOKIE = "my_cookie";
235 MY_VAR = "my_var";
236 };
237 serviceConfig = {
238 Type = "exec";
239 DynamicUser = true;
240 WorkingDirectory = working_directory;
241 # Implied by DynamicUser, but just to emphasize due to RELEASE_TMP
242 PrivateTmp = true;
243 ExecStart = ''
244 ${release}/bin/${release_name} start
245 '';
246 ExecStop = ''
247 ${release}/bin/${release_name} stop
248 '';
249 ExecReload = ''
250 ${release}/bin/${release_name} restart
251 '';
252 Restart = "on-failure";
253 RestartSec = 5;
254 StartLimitBurst = 3;
255 StartLimitInterval = 10;
256 };
257 # disksup requires bash
258 path = [ pkgs.bash ];
259 };
260
261 # in case you have migration scripts or you want to use a remote shell
262 environment.systemPackages = [ release ];
263}
264```
265
266## How to Develop {#how-to-develop}
267
268### Creating a Shell {#creating-a-shell}
269
270Usually, we need to create a `shell.nix` file and do our development inside of the environment specified therein. Just install your version of Erlang and any other interpreters, and then use your normal build tools. As an example with Elixir:
271
272```nix
273{ pkgs ? import <nixpkgs> {} }:
274
275with pkgs;
276let
277 elixir = beam.packages.erlang_24.elixir_1_12;
278in
279mkShell {
280 buildInputs = [ elixir ];
281}
282```
283
284### Using an overlay {#beam-using-overlays}
285
286If you need to use an overlay to change some attributes of a derivation, e.g. if you need a bugfix from a version that is not yet available in nixpkgs, you can override attributes such as `version` (and the corresponding `hash`) and then use this overlay in your development environment:
287
288#### `shell.nix` {#beam-using-overlays-shell.nix}
289
290```nix
291let
292 elixir_1_13_1_overlay = (self: super: {
293 elixir_1_13 = super.elixir_1_13.override {
294 version = "1.13.1";
295 sha256 = "sha256-t0ic1LcC7EV3avWGdR7VbyX7pGDpnJSW1ZvwvQUPC3w=";
296 };
297 });
298 pkgs = import <nixpkgs> { overlays = [ elixir_1_13_1_overlay ]; };
299in
300with pkgs;
301mkShell {
302 buildInputs = [
303 elixir_1_13
304 ];
305}
306```
307
308#### Elixir - Phoenix project {#elixir---phoenix-project}
309
310Here is an example `shell.nix`.
311
312```nix
313with import <nixpkgs> { };
314
315let
316 # define packages to install
317 basePackages = [
318 git
319 # replace with beam.packages.erlang.elixir_1_13 if you need
320 beam.packages.erlang.elixir
321 nodejs
322 postgresql_14
323 # only used for frontend dependencies
324 # you are free to use yarn2nix as well
325 nodePackages.node2nix
326 # formatting js file
327 nodePackages.prettier
328 ];
329
330 inputs = basePackages ++ lib.optionals stdenv.isLinux [ inotify-tools ]
331 ++ lib.optionals stdenv.isDarwin
332 (with darwin.apple_sdk.frameworks; [ CoreFoundation CoreServices ]);
333
334 # define shell startup command
335 hooks = ''
336 # this allows mix to work on the local directory
337 mkdir -p .nix-mix .nix-hex
338 export MIX_HOME=$PWD/.nix-mix
339 export HEX_HOME=$PWD/.nix-mix
340 # make hex from Nixpkgs available
341 # `mix local.hex` will install hex into MIX_HOME and should take precedence
342 export MIX_PATH="${beam.packages.erlang.hex}/lib/erlang/lib/hex/ebin"
343 export PATH=$MIX_HOME/bin:$HEX_HOME/bin:$PATH
344 export LANG=C.UTF-8
345 # keep your shell history in iex
346 export ERL_AFLAGS="-kernel shell_history enabled"
347
348 # postges related
349 # keep all your db data in a folder inside the project
350 export PGDATA="$PWD/db"
351
352 # phoenix related env vars
353 export POOL_SIZE=15
354 export DB_URL="postgresql://postgres:postgres@localhost:5432/db"
355 export PORT=4000
356 export MIX_ENV=dev
357 # add your project env vars here, word readable in the nix store.
358 export ENV_VAR="your_env_var"
359 '';
360
361in mkShell {
362 buildInputs = inputs;
363 shellHook = hooks;
364}
365```
366
367Initializing the project will require the following steps:
368
369- create the db directory `initdb ./db` (inside your mix project folder)
370- create the postgres user `createuser postgres -ds`
371- create the db `createdb db`
372- start the postgres instance `pg_ctl -l "$PGDATA/server.log" start`
373- add the `/db` folder to your `.gitignore`
374- you can start your phoenix server and get a shell with `iex -S mix phx.server`