Thicket data repository for the EEG
anil.recoil.org
1{
2 "id": "https://ryan.freumh.org/opam-nix.html",
3 "title": "Opam's Nix system dependency mechanism",
4 "link": "https://ryan.freumh.org/opam-nix.html",
5 "updated": "2025-05-02T00:00:00",
6 "published": "2025-04-25T00:00:00",
7 "summary": "<div>\n \n <span>Published 25 Apr 2025.</span>\n \n \n <span>Last update 2 May 2025.</span>\n \n </div>\n \n <div> Tags: <a href=\"/projects.html\" title=\"All pages tagged 'projects'.\">projects</a>. </div>\n \n \n\n <p><span>On 22 Apr 2022, three years ago, I opened an issue\nin the OCaml package manager, opam, ‘<a href=\"https://github.com/ocaml/opam/issues/5124\">depext does not support\nnixOS</a>’. Last week, my pull request fixing this got <a href=\"https://github.com/ocaml/opam/pull/5982\">merged</a>!</span></p>\n<h2>Let’s Encrypt Example</h2>\n<p><span>Before, if we tried installing\nan OCaml package with a system dependency we would run into:</span></p>\n<pre><code>$ opam --version\n2.3.0\n$ opam install letsencrypt\n[NOTE] External dependency handling not supported for OS family 'nixos'.\n You can disable this check using 'opam option --global depext=false'\n[NOTE] It seems you have not updated your repositories for a while. Consider updating them with:\n opam update\n\nThe following actions will be performed:\n=== install 41 packages\n...\n ∗ conf-gmp 4 [required by zarith]\n ∗ conf-pkg-config 4 [required by zarith]\n ∗ letsencrypt 1.1.0\n ∗ mirage-crypto-pk 2.0.0 [required by letsencrypt]\n ∗ zarith 1.14 [required by mirage-crypto-pk]\n\nProceed with ∗ 41 installations? [y/n] y\n\n<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>\n⬇ retrieved asn1-combinators.0.3.2 (cached)\n⬇ retrieved base64.3.5.1 (cached)\n⬇ retrieved conf-gmp.4 (cached)\n...\n[ERROR] The compilation of conf-gmp.4 failed at "sh -exc cc -c $CFLAGS -I/usr/local/include test.c".\n...\n\n#=== ERROR while compiling conf-gmp.4 =========================================#\n# context 2.3.0 | linux/x86_64 | ocaml-base-compiler.5.3.0 | https://opam.ocaml.org#4d8fa0fb8fce3b6c8b06f29ebcfa844c292d4f3e\n# path ~/.opam/ocaml-base-compiler.5.3.0/.opam-switch/build/conf-gmp.4\n# command ~/.opam/opam-init/hooks/sandbox.sh build sh -exc cc -c $CFLAGS -I/usr/local/include test.c\n# exit-code 1\n# env-file ~/.opam/log/conf-gmp-1821939-442af5.env\n# output-file ~/.opam/log/conf-gmp-1821939-442af5.out\n### output ###\n# + cc -c -I/usr/local/include test.c\n# test.c:1:10: fatal error: gmp.h: No such file or directory\n# 1 | #include <gmp.h>\n# | ^~~~~~~\n# compilation terminated.\n\n<><> Error report <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>\n┌─ The following actions failed\n│ λ build conf-gmp 4\n└─\n...\n</code></pre>\n<p><span>Now, it looks like:</span></p>\n<pre><code>$ opam --version\n2.4.0~alpha1\n$ opam install letsencrypt\nThe following actions will be performed:\n=== install 41 packages\n...\n ∗ conf-gmp 4 [required by zarith]\n ∗ conf-pkg-config 4 [required by zarith]\n ∗ letsencrypt 1.1.0\n ∗ mirage-crypto-pk 2.0.0 [required by letsencrypt]\n ∗ zarith 1.14 [required by mirage-crypto-pk]\n\nProceed with ∗ 41 installations? [Y/n] y\n\nThe following system packages will first need to be installed:\n gmp pkg-config\n\n<><> Handling external dependencies <><><><><><><><><><><><><><><><><><><><><><>\n\nopam believes some required external dependencies are missing. opam can:\n> 1. Run nix-build to install them (may need root/sudo access)\n 2. Display the recommended nix-build command and wait while you run it manually (e.g. in another\n terminal)\n 3. Continue anyway, and, upon success, permanently register that this external dependency is present, but\n not detectable\n 4. Abort the installation\n\n[1/2/3/4] 1\n\n+ /run/current-system/sw/bin/nix-build "/home/ryan/.opam/ocaml-base-compiler.5.3.0/.opam-switch/env.nix" "--out-link" "/home/ryan/.opam/ocaml-base-compiler.5.3.0/.opam-switch/nix.env"\n- this derivation will be built:\n- /nix/store/7ym3yz334i01zr5xk7d1bvdbv34ipa3a-opam-nix-env.drv\n- building '/nix/store/7ym3yz334i01zr5xk7d1bvdbv34ipa3a-opam-nix-env.drv'...\n- Running phase: buildPhase\n- /nix/store/sjvwj70igi44svwj32l8mk9v9g6rrqr4-opam-nix-env\n\n<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>\n...\n⬇ retrieved conf-gmp.4 (cached)\n⬇ retrieved conf-gmp-powm-sec.3 (cached)\n∗ installed conf-pkg-config.4\n∗ installed conf-gmp.4\n⬇ retrieved letsencrypt.1.1.0 (cached)\n⬇ retrieved mirage-crypto.2.0.0, mirage-crypto-ec.2.0.0, mirage-crypto-pk.2.0.0, mirage-crypto-rng.2.0.0 (cached)\n⬇ retrieved zarith.1.14 (cached)\n∗ installed zarith.1.14\n∗ installed mirage-crypto-pk.2.0.0\n∗ installed letsencrypt.1.1.0\nDone.\n# To update the current shell environment, run: eval $(opam env)\n</code></pre>\n<h2>Implementation</h2>\n<p><span>Some background: opam has an ‘<a href=\"https://opam.ocaml.org/doc/Manual.html#opamfield-depexts\">external\ndependency</a>’ (depext) system where packages can declare dependencies\non packages that are provided by Operating System package managers\nrather than opam. One such depext is the <a href=\"https://gmplib.org/\">GMP</a> C library used by <a href=\"https://github.com/ocaml/Zarith\">Zarith</a>, which can be\ninstalled on Debian with <code>apt install libgmp-dev</code>. The opam repository has\nvirtual <code>conf-*</code> packages which unify\ndependencies across ecosystems, so <code>conf-gmp</code> contains:</span></p>\n<pre><code>depexts: [\n ["libgmp-dev"] {os-family = "debian"}\n ["libgmp-dev"] {os-family = "ubuntu"}\n ["gmp"] {os = "macos" & os-distribution = "homebrew"}\n ["gmp"] {os-distribution = "macports" & os = "macos"}\n ...\n ["gmp"] {os-distribution = "nixos"}\n]\n</code></pre>\n<p><span>Where depexts entries are <a href=\"https://opam.ocaml.org/doc/Manual.html#Filters\">filtered</a>\naccording to variables describing the system package manager.</span></p>\n<p><span>However, <a href=\"nix.html\">Nix</a>OS\nhas a <a href=\"https://discourse.nixos.org/t/query-all-pnames-in-nixpkgs-with-flakes/22879/3\">rather\ndifferent notion of installation</a> than other Linux distributions.\nSpecifically, environment variables for linkers to find libraries are\nset in a Nix derivation, not when installing a package to the system. So\n<a href=\"https://github.com/ocaml/opam/pull/5332\">attempts</a> to invoke\n<code>nix-env</code> to provide Nix system dependencies\nwere limited to executables.</span></p>\n<p><span>Instead, to use GMP, one had to\ninvoke <code>nix-shell -p gmp</code> before invoking\nthe build system. This is suboptimal for two reasons:</span></p>\n<ol>\n<li>It requires manual resolution of system dependencies.</li>\n<li>The resulting binary will contain a reference to a path in the Nix\nstore which isn’t part of a garbage collection (GC) root, so on the next\nNix GC the binary will stop working.</li>\n</ol>\n<p><span>The obvious fix for the latter is to\nbuild the binary as a Nix derivation, making it a GC root, which is what\n<a href=\"https://github.com/tweag/opam-nix\">opam-nix</a> supports. It\nuses opam to solve dependencies inside a Nix derivation, uses Nix’s <a href=\"https://nix.dev/manual/nix/2.28/language/import-from-derivation\">Import\nFrom Derivation</a> to see the resolved dependencies, and creates Nix\nderivations for the resulting dependencies. Using the depexts filtered\nwith <code>os-distribution = \"nixos\"</code> opam-nix is\nable to provide system dependencies from Nixpkgs.</span></p>\n<p><span>While working with opam-nix when\nbuilding <a href=\"hillingar.html\">Hillingar</a> I found it to be great\nfor deploying OCaml programs on NixOS systems (e.g. <a href=\"eon.html\">Eon</a>), but it was slow and unergonomic for\ndevelopment. Every time a dependency is added or changed, an expensive\nNix rebuild is required; it’s a lot faster just to work with\nOpam.</span></p>\n<p><span>On 8 Apr 2024 I got funding for a\nproject that included adding depext support for NixOS to opam. There\nwere a few <a href=\"https://github.com/ocaml/opam/pull/5942\">false</a>\n<a href=\"https://github.com/RyanGibb/nix.opam\">starts</a> along the way\nbut eventually I implemented a <a href=\"https://github.com/ocaml/opam/pull/5982\">depext mechanism that\nmanages a <code>nix-shell</code>-like environment</a>,\nsetting environment variables with Opam to make system dependencies\n(depexts) available with Nix. We create a Nix derivation\nlike,</span></p>\n<div><pre><code><span><a href=\"#cb4-1\"></a><span>{</span> <span>pkgs</span> <span>?</span> <span>import</span> <nixpkgs> <span>{}</span> <span>}</span>:</span>\n<span><a href=\"#cb4-2\"></a><span>with</span> pkgs<span>;</span></span>\n<span><a href=\"#cb4-3\"></a>stdenv.mkDerivation <span>{</span></span>\n<span><a href=\"#cb4-4\"></a> <span>name</span> <span>=</span> <span>"opam-nix-env"</span><span>;</span></span>\n<span><a href=\"#cb4-5\"></a> <span>nativeBuildInputs</span> <span>=</span> <span>with</span> buildPackages<span>;</span> <span>[</span> pkg-config gmp <span>];</span></span>\n<span><a href=\"#cb4-6\"></a></span>\n<span><a href=\"#cb4-7\"></a> <span>phases</span> <span>=</span> <span>[</span> <span>"buildPhase"</span> <span>];</span></span>\n<span><a href=\"#cb4-8\"></a></span>\n<span><a href=\"#cb4-9\"></a> <span>buildPhase</span> <span>=</span> <span>''</span></span>\n<span><a href=\"#cb4-10\"></a><span>while IFS='=' read -r var value; do</span></span>\n<span><a href=\"#cb4-11\"></a><span> escaped="</span><span>''$</span><span>(echo "$value" | sed -e 's/^$/@/' -e 's/ /\\\\ /g')"</span></span>\n<span><a href=\"#cb4-12\"></a><span> echo "$var\t=\t$escaped\tNix" >> "$out"</span></span>\n<span><a href=\"#cb4-13\"></a><span>done < <(env \\</span></span>\n<span><a href=\"#cb4-14\"></a><span> -u BASHOPTS \\</span></span>\n<span><a href=\"#cb4-15\"></a><span> -u HOME \\</span></span>\n<span><a href=\"#cb4-16\"></a><span> -u NIX_BUILD_TOP \\</span></span>\n<span><a href=\"#cb4-17\"></a><span> -u NIX_ENFORCE_PURITY \\</span></span>\n<span><a href=\"#cb4-18\"></a><span> -u NIX_LOG_FD \\</span></span>\n<span><a href=\"#cb4-19\"></a><span> -u NIX_REMOTE \\</span></span>\n<span><a href=\"#cb4-20\"></a><span> -u PPID \\</span></span>\n<span><a href=\"#cb4-21\"></a><span> -u SHELLOPTS \\</span></span>\n<span><a href=\"#cb4-22\"></a><span> -u SSL_CERT_FILE \\</span></span>\n<span><a href=\"#cb4-23\"></a><span> -u TEMP \\</span></span>\n<span><a href=\"#cb4-24\"></a><span> -u TEMPDIR \\</span></span>\n<span><a href=\"#cb4-25\"></a><span> -u TERM \\</span></span>\n<span><a href=\"#cb4-26\"></a><span> -u TMP \\</span></span>\n<span><a href=\"#cb4-27\"></a><span> -u TMPDIR \\</span></span>\n<span><a href=\"#cb4-28\"></a><span> -u TZ \\</span></span>\n<span><a href=\"#cb4-29\"></a><span> -u UID \\</span></span>\n<span><a href=\"#cb4-30\"></a><span> -u PATH \\</span></span>\n<span><a href=\"#cb4-31\"></a><span> -u XDG_DATA_DIRS \\</span></span>\n<span><a href=\"#cb4-32\"></a><span> -u self-referential \\</span></span>\n<span><a href=\"#cb4-33\"></a><span> -u excluded_vars \\</span></span>\n<span><a href=\"#cb4-34\"></a><span> -u excluded_pattern \\</span></span>\n<span><a href=\"#cb4-35\"></a><span> -u phases \\</span></span>\n<span><a href=\"#cb4-36\"></a><span> -u buildPhase \\</span></span>\n<span><a href=\"#cb4-37\"></a><span> -u outputs)</span></span>\n<span><a href=\"#cb4-38\"></a></span>\n<span><a href=\"#cb4-39\"></a><span>echo "PATH\t+=\t$PATH\tNix" >> "$out"</span></span>\n<span><a href=\"#cb4-40\"></a><span>echo "XDG_DATA_DIRS\t+=\t$XDG_DATA_DIRS\tNix" >> "$out"</span></span>\n<span><a href=\"#cb4-41\"></a><span> ''</span><span>;</span></span>\n<span><a href=\"#cb4-42\"></a></span>\n<span><a href=\"#cb4-43\"></a> <span>preferLocalBuild</span> <span>=</span> <span>true</span><span>;</span></span>\n<span><a href=\"#cb4-44\"></a><span>}</span></span></code></pre></div>\n<p><span>Which is very similar to how <code>nix-shell</code> and its successor <code>nix develop</code> work under the hood, and we get the\nlist of variables to <a href=\"https://github.com/NixOS/nix/blob/e4bda20918ad2af690c2e938211a7d362548e403/src/nix/develop.cc#L308-L325\">exclude</a>\nand <a href=\"https://github.com/NixOS/nix/blob/e4bda20918ad2af690c2e938211a7d362548e403/src/nix/develop.cc#L347-L353\">append</a>\ntoo from the <code>nix develop</code> source. We build\nthis Nix derivation to output a file in Opam’s environment variable\nformat containing variables to make depexts available. This environment\nfile is a Nix store root, so its dependencies won’t be garbage collected\nby Nix until the file is removed. This depext mechanism is quite\ndifferent to the imperative model most other system package managers\nused, so required a fair amount of refactoring to be plumbed through the\ncodebase.</span></p>\n<p><span>A really cool aspect of this depext\nmechanism is that it doesn’t interfere with the system environment, so\nit allows totally isolated environments for different projects. This\ncould be useful to use on even non-NixOS systems as a result.</span></p>\n<p><span>Opam’s Nix depext mechanism has been\nmerged and released in Opam 2.4~alpha1, which you can use on NixOS with\n<a href=\"https://github.com/RyanGibb/nixos/blob/41590b9ee0e8407cf5a274c8e1af7decd993a824/flake.nix#L70-L77\">this</a>\noverlay:</span></p>\n<div><pre><code><span><a href=\"#cb5-1\"></a>opam = final.overlay<span>-</span>unstable.opam.overrideAttrs <span>(</span><span>_</span><span>:</span> <span>rec</span> <span>{</span></span>\n<span><a href=\"#cb5-2\"></a> <span>version</span> <span>=</span> <span>"2.4.0-alpha1"</span><span>;</span></span>\n<span><a href=\"#cb5-3\"></a> <span>src</span> <span>=</span> final.fetchurl <span>{</span></span>\n<span><a href=\"#cb5-4\"></a> <span>url</span> <span>=</span> <span>"https://github.com/ocaml/opam/releases/download/</span><span>${</span>version<span>}</span><span>/opam-full-</span><span>${</span>version<span>}</span><span>.tar.gz"</span><span>;</span></span>\n<span><a href=\"#cb5-5\"></a> <span>sha256</span> <span>=</span> <span>"sha256-kRGh8K5sMvmbJtSAEEPIOsim8uUUhrw11I+vVd/nnx4="</span><span>;</span></span>\n<span><a href=\"#cb5-6\"></a> <span>};</span></span>\n<span><a href=\"#cb5-7\"></a> <span>patches</span> <span>=</span> <span>[</span> <span>./pkgs/opam-shebangs.patch</span> <span>];</span></span>\n<span><a href=\"#cb5-8\"></a><span>})</span>;</span></code></pre></div>\n<p><span>And can be used from my repository\ndirectly:</span></p>\n<div><pre><code><span><a href=\"#cb6-1\"></a><span>$</span> nix shell github:RyanGibb/nixos#legacyPackages.x86_64-linux.nixpkgs.opam</span></code></pre></div>\n<p><span>Another part of this project was\nbridging version solving with Nix<a href=\"#fn1\">1</a> in <a href=\"https://github.com/RyanGibb/opam-nix-repository\">opam-nix-repository</a>\nwhich has continued into the <a href=\"enki.html\">Enki</a>\nproject.</span></p>\n<p><span>Thanks to David, Kate, and Raja for\nall their help, and to Jane Street for funding this work.</span></p>\n\n\n\n\n<ol>\n<li><p><span><a href=\"https://github.com/NixOS/nixpkgs/issues/9682\">Which lacks version\nsolving</a>.</span><a href=\"#fnref1\">↩︎</a></p></li>\n</ol>",
8 "content": "<div>\n \n <span>Published 25 Apr 2025.</span>\n \n \n <span>Last update 2 May 2025.</span>\n \n </div>\n \n <div> Tags: <a href=\"/projects.html\" title=\"All pages tagged 'projects'.\">projects</a>. </div>\n \n \n\n <p><span>On 22 Apr 2022, three years ago, I opened an issue\nin the OCaml package manager, opam, ‘<a href=\"https://github.com/ocaml/opam/issues/5124\">depext does not support\nnixOS</a>’. Last week, my pull request fixing this got <a href=\"https://github.com/ocaml/opam/pull/5982\">merged</a>!</span></p>\n<h2>Let’s Encrypt Example</h2>\n<p><span>Before, if we tried installing\nan OCaml package with a system dependency we would run into:</span></p>\n<pre><code>$ opam --version\n2.3.0\n$ opam install letsencrypt\n[NOTE] External dependency handling not supported for OS family 'nixos'.\n You can disable this check using 'opam option --global depext=false'\n[NOTE] It seems you have not updated your repositories for a while. Consider updating them with:\n opam update\n\nThe following actions will be performed:\n=== install 41 packages\n...\n ∗ conf-gmp 4 [required by zarith]\n ∗ conf-pkg-config 4 [required by zarith]\n ∗ letsencrypt 1.1.0\n ∗ mirage-crypto-pk 2.0.0 [required by letsencrypt]\n ∗ zarith 1.14 [required by mirage-crypto-pk]\n\nProceed with ∗ 41 installations? [y/n] y\n\n<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>\n⬇ retrieved asn1-combinators.0.3.2 (cached)\n⬇ retrieved base64.3.5.1 (cached)\n⬇ retrieved conf-gmp.4 (cached)\n...\n[ERROR] The compilation of conf-gmp.4 failed at "sh -exc cc -c $CFLAGS -I/usr/local/include test.c".\n...\n\n#=== ERROR while compiling conf-gmp.4 =========================================#\n# context 2.3.0 | linux/x86_64 | ocaml-base-compiler.5.3.0 | https://opam.ocaml.org#4d8fa0fb8fce3b6c8b06f29ebcfa844c292d4f3e\n# path ~/.opam/ocaml-base-compiler.5.3.0/.opam-switch/build/conf-gmp.4\n# command ~/.opam/opam-init/hooks/sandbox.sh build sh -exc cc -c $CFLAGS -I/usr/local/include test.c\n# exit-code 1\n# env-file ~/.opam/log/conf-gmp-1821939-442af5.env\n# output-file ~/.opam/log/conf-gmp-1821939-442af5.out\n### output ###\n# + cc -c -I/usr/local/include test.c\n# test.c:1:10: fatal error: gmp.h: No such file or directory\n# 1 | #include <gmp.h>\n# | ^~~~~~~\n# compilation terminated.\n\n<><> Error report <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>\n┌─ The following actions failed\n│ λ build conf-gmp 4\n└─\n...\n</code></pre>\n<p><span>Now, it looks like:</span></p>\n<pre><code>$ opam --version\n2.4.0~alpha1\n$ opam install letsencrypt\nThe following actions will be performed:\n=== install 41 packages\n...\n ∗ conf-gmp 4 [required by zarith]\n ∗ conf-pkg-config 4 [required by zarith]\n ∗ letsencrypt 1.1.0\n ∗ mirage-crypto-pk 2.0.0 [required by letsencrypt]\n ∗ zarith 1.14 [required by mirage-crypto-pk]\n\nProceed with ∗ 41 installations? [Y/n] y\n\nThe following system packages will first need to be installed:\n gmp pkg-config\n\n<><> Handling external dependencies <><><><><><><><><><><><><><><><><><><><><><>\n\nopam believes some required external dependencies are missing. opam can:\n> 1. Run nix-build to install them (may need root/sudo access)\n 2. Display the recommended nix-build command and wait while you run it manually (e.g. in another\n terminal)\n 3. Continue anyway, and, upon success, permanently register that this external dependency is present, but\n not detectable\n 4. Abort the installation\n\n[1/2/3/4] 1\n\n+ /run/current-system/sw/bin/nix-build "/home/ryan/.opam/ocaml-base-compiler.5.3.0/.opam-switch/env.nix" "--out-link" "/home/ryan/.opam/ocaml-base-compiler.5.3.0/.opam-switch/nix.env"\n- this derivation will be built:\n- /nix/store/7ym3yz334i01zr5xk7d1bvdbv34ipa3a-opam-nix-env.drv\n- building '/nix/store/7ym3yz334i01zr5xk7d1bvdbv34ipa3a-opam-nix-env.drv'...\n- Running phase: buildPhase\n- /nix/store/sjvwj70igi44svwj32l8mk9v9g6rrqr4-opam-nix-env\n\n<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>\n...\n⬇ retrieved conf-gmp.4 (cached)\n⬇ retrieved conf-gmp-powm-sec.3 (cached)\n∗ installed conf-pkg-config.4\n∗ installed conf-gmp.4\n⬇ retrieved letsencrypt.1.1.0 (cached)\n⬇ retrieved mirage-crypto.2.0.0, mirage-crypto-ec.2.0.0, mirage-crypto-pk.2.0.0, mirage-crypto-rng.2.0.0 (cached)\n⬇ retrieved zarith.1.14 (cached)\n∗ installed zarith.1.14\n∗ installed mirage-crypto-pk.2.0.0\n∗ installed letsencrypt.1.1.0\nDone.\n# To update the current shell environment, run: eval $(opam env)\n</code></pre>\n<h2>Implementation</h2>\n<p><span>Some background: opam has an ‘<a href=\"https://opam.ocaml.org/doc/Manual.html#opamfield-depexts\">external\ndependency</a>’ (depext) system where packages can declare dependencies\non packages that are provided by Operating System package managers\nrather than opam. One such depext is the <a href=\"https://gmplib.org/\">GMP</a> C library used by <a href=\"https://github.com/ocaml/Zarith\">Zarith</a>, which can be\ninstalled on Debian with <code>apt install libgmp-dev</code>. The opam repository has\nvirtual <code>conf-*</code> packages which unify\ndependencies across ecosystems, so <code>conf-gmp</code> contains:</span></p>\n<pre><code>depexts: [\n ["libgmp-dev"] {os-family = "debian"}\n ["libgmp-dev"] {os-family = "ubuntu"}\n ["gmp"] {os = "macos" & os-distribution = "homebrew"}\n ["gmp"] {os-distribution = "macports" & os = "macos"}\n ...\n ["gmp"] {os-distribution = "nixos"}\n]\n</code></pre>\n<p><span>Where depexts entries are <a href=\"https://opam.ocaml.org/doc/Manual.html#Filters\">filtered</a>\naccording to variables describing the system package manager.</span></p>\n<p><span>However, <a href=\"nix.html\">Nix</a>OS\nhas a <a href=\"https://discourse.nixos.org/t/query-all-pnames-in-nixpkgs-with-flakes/22879/3\">rather\ndifferent notion of installation</a> than other Linux distributions.\nSpecifically, environment variables for linkers to find libraries are\nset in a Nix derivation, not when installing a package to the system. So\n<a href=\"https://github.com/ocaml/opam/pull/5332\">attempts</a> to invoke\n<code>nix-env</code> to provide Nix system dependencies\nwere limited to executables.</span></p>\n<p><span>Instead, to use GMP, one had to\ninvoke <code>nix-shell -p gmp</code> before invoking\nthe build system. This is suboptimal for two reasons:</span></p>\n<ol>\n<li>It requires manual resolution of system dependencies.</li>\n<li>The resulting binary will contain a reference to a path in the Nix\nstore which isn’t part of a garbage collection (GC) root, so on the next\nNix GC the binary will stop working.</li>\n</ol>\n<p><span>The obvious fix for the latter is to\nbuild the binary as a Nix derivation, making it a GC root, which is what\n<a href=\"https://github.com/tweag/opam-nix\">opam-nix</a> supports. It\nuses opam to solve dependencies inside a Nix derivation, uses Nix’s <a href=\"https://nix.dev/manual/nix/2.28/language/import-from-derivation\">Import\nFrom Derivation</a> to see the resolved dependencies, and creates Nix\nderivations for the resulting dependencies. Using the depexts filtered\nwith <code>os-distribution = \"nixos\"</code> opam-nix is\nable to provide system dependencies from Nixpkgs.</span></p>\n<p><span>While working with opam-nix when\nbuilding <a href=\"hillingar.html\">Hillingar</a> I found it to be great\nfor deploying OCaml programs on NixOS systems (e.g. <a href=\"eon.html\">Eon</a>), but it was slow and unergonomic for\ndevelopment. Every time a dependency is added or changed, an expensive\nNix rebuild is required; it’s a lot faster just to work with\nOpam.</span></p>\n<p><span>On 8 Apr 2024 I got funding for a\nproject that included adding depext support for NixOS to opam. There\nwere a few <a href=\"https://github.com/ocaml/opam/pull/5942\">false</a>\n<a href=\"https://github.com/RyanGibb/nix.opam\">starts</a> along the way\nbut eventually I implemented a <a href=\"https://github.com/ocaml/opam/pull/5982\">depext mechanism that\nmanages a <code>nix-shell</code>-like environment</a>,\nsetting environment variables with Opam to make system dependencies\n(depexts) available with Nix. We create a Nix derivation\nlike,</span></p>\n<div><pre><code><span><a href=\"#cb4-1\"></a><span>{</span> <span>pkgs</span> <span>?</span> <span>import</span> <nixpkgs> <span>{}</span> <span>}</span>:</span>\n<span><a href=\"#cb4-2\"></a><span>with</span> pkgs<span>;</span></span>\n<span><a href=\"#cb4-3\"></a>stdenv.mkDerivation <span>{</span></span>\n<span><a href=\"#cb4-4\"></a> <span>name</span> <span>=</span> <span>"opam-nix-env"</span><span>;</span></span>\n<span><a href=\"#cb4-5\"></a> <span>nativeBuildInputs</span> <span>=</span> <span>with</span> buildPackages<span>;</span> <span>[</span> pkg-config gmp <span>];</span></span>\n<span><a href=\"#cb4-6\"></a></span>\n<span><a href=\"#cb4-7\"></a> <span>phases</span> <span>=</span> <span>[</span> <span>"buildPhase"</span> <span>];</span></span>\n<span><a href=\"#cb4-8\"></a></span>\n<span><a href=\"#cb4-9\"></a> <span>buildPhase</span> <span>=</span> <span>''</span></span>\n<span><a href=\"#cb4-10\"></a><span>while IFS='=' read -r var value; do</span></span>\n<span><a href=\"#cb4-11\"></a><span> escaped="</span><span>''$</span><span>(echo "$value" | sed -e 's/^$/@/' -e 's/ /\\\\ /g')"</span></span>\n<span><a href=\"#cb4-12\"></a><span> echo "$var\t=\t$escaped\tNix" >> "$out"</span></span>\n<span><a href=\"#cb4-13\"></a><span>done < <(env \\</span></span>\n<span><a href=\"#cb4-14\"></a><span> -u BASHOPTS \\</span></span>\n<span><a href=\"#cb4-15\"></a><span> -u HOME \\</span></span>\n<span><a href=\"#cb4-16\"></a><span> -u NIX_BUILD_TOP \\</span></span>\n<span><a href=\"#cb4-17\"></a><span> -u NIX_ENFORCE_PURITY \\</span></span>\n<span><a href=\"#cb4-18\"></a><span> -u NIX_LOG_FD \\</span></span>\n<span><a href=\"#cb4-19\"></a><span> -u NIX_REMOTE \\</span></span>\n<span><a href=\"#cb4-20\"></a><span> -u PPID \\</span></span>\n<span><a href=\"#cb4-21\"></a><span> -u SHELLOPTS \\</span></span>\n<span><a href=\"#cb4-22\"></a><span> -u SSL_CERT_FILE \\</span></span>\n<span><a href=\"#cb4-23\"></a><span> -u TEMP \\</span></span>\n<span><a href=\"#cb4-24\"></a><span> -u TEMPDIR \\</span></span>\n<span><a href=\"#cb4-25\"></a><span> -u TERM \\</span></span>\n<span><a href=\"#cb4-26\"></a><span> -u TMP \\</span></span>\n<span><a href=\"#cb4-27\"></a><span> -u TMPDIR \\</span></span>\n<span><a href=\"#cb4-28\"></a><span> -u TZ \\</span></span>\n<span><a href=\"#cb4-29\"></a><span> -u UID \\</span></span>\n<span><a href=\"#cb4-30\"></a><span> -u PATH \\</span></span>\n<span><a href=\"#cb4-31\"></a><span> -u XDG_DATA_DIRS \\</span></span>\n<span><a href=\"#cb4-32\"></a><span> -u self-referential \\</span></span>\n<span><a href=\"#cb4-33\"></a><span> -u excluded_vars \\</span></span>\n<span><a href=\"#cb4-34\"></a><span> -u excluded_pattern \\</span></span>\n<span><a href=\"#cb4-35\"></a><span> -u phases \\</span></span>\n<span><a href=\"#cb4-36\"></a><span> -u buildPhase \\</span></span>\n<span><a href=\"#cb4-37\"></a><span> -u outputs)</span></span>\n<span><a href=\"#cb4-38\"></a></span>\n<span><a href=\"#cb4-39\"></a><span>echo "PATH\t+=\t$PATH\tNix" >> "$out"</span></span>\n<span><a href=\"#cb4-40\"></a><span>echo "XDG_DATA_DIRS\t+=\t$XDG_DATA_DIRS\tNix" >> "$out"</span></span>\n<span><a href=\"#cb4-41\"></a><span> ''</span><span>;</span></span>\n<span><a href=\"#cb4-42\"></a></span>\n<span><a href=\"#cb4-43\"></a> <span>preferLocalBuild</span> <span>=</span> <span>true</span><span>;</span></span>\n<span><a href=\"#cb4-44\"></a><span>}</span></span></code></pre></div>\n<p><span>Which is very similar to how <code>nix-shell</code> and its successor <code>nix develop</code> work under the hood, and we get the\nlist of variables to <a href=\"https://github.com/NixOS/nix/blob/e4bda20918ad2af690c2e938211a7d362548e403/src/nix/develop.cc#L308-L325\">exclude</a>\nand <a href=\"https://github.com/NixOS/nix/blob/e4bda20918ad2af690c2e938211a7d362548e403/src/nix/develop.cc#L347-L353\">append</a>\ntoo from the <code>nix develop</code> source. We build\nthis Nix derivation to output a file in Opam’s environment variable\nformat containing variables to make depexts available. This environment\nfile is a Nix store root, so its dependencies won’t be garbage collected\nby Nix until the file is removed. This depext mechanism is quite\ndifferent to the imperative model most other system package managers\nused, so required a fair amount of refactoring to be plumbed through the\ncodebase.</span></p>\n<p><span>A really cool aspect of this depext\nmechanism is that it doesn’t interfere with the system environment, so\nit allows totally isolated environments for different projects. This\ncould be useful to use on even non-NixOS systems as a result.</span></p>\n<p><span>Opam’s Nix depext mechanism has been\nmerged and released in Opam 2.4~alpha1, which you can use on NixOS with\n<a href=\"https://github.com/RyanGibb/nixos/blob/41590b9ee0e8407cf5a274c8e1af7decd993a824/flake.nix#L70-L77\">this</a>\noverlay:</span></p>\n<div><pre><code><span><a href=\"#cb5-1\"></a>opam = final.overlay<span>-</span>unstable.opam.overrideAttrs <span>(</span><span>_</span><span>:</span> <span>rec</span> <span>{</span></span>\n<span><a href=\"#cb5-2\"></a> <span>version</span> <span>=</span> <span>"2.4.0-alpha1"</span><span>;</span></span>\n<span><a href=\"#cb5-3\"></a> <span>src</span> <span>=</span> final.fetchurl <span>{</span></span>\n<span><a href=\"#cb5-4\"></a> <span>url</span> <span>=</span> <span>"https://github.com/ocaml/opam/releases/download/</span><span>${</span>version<span>}</span><span>/opam-full-</span><span>${</span>version<span>}</span><span>.tar.gz"</span><span>;</span></span>\n<span><a href=\"#cb5-5\"></a> <span>sha256</span> <span>=</span> <span>"sha256-kRGh8K5sMvmbJtSAEEPIOsim8uUUhrw11I+vVd/nnx4="</span><span>;</span></span>\n<span><a href=\"#cb5-6\"></a> <span>};</span></span>\n<span><a href=\"#cb5-7\"></a> <span>patches</span> <span>=</span> <span>[</span> <span>./pkgs/opam-shebangs.patch</span> <span>];</span></span>\n<span><a href=\"#cb5-8\"></a><span>})</span>;</span></code></pre></div>\n<p><span>And can be used from my repository\ndirectly:</span></p>\n<div><pre><code><span><a href=\"#cb6-1\"></a><span>$</span> nix shell github:RyanGibb/nixos#legacyPackages.x86_64-linux.nixpkgs.opam</span></code></pre></div>\n<p><span>Another part of this project was\nbridging version solving with Nix<a href=\"#fn1\">1</a> in <a href=\"https://github.com/RyanGibb/opam-nix-repository\">opam-nix-repository</a>\nwhich has continued into the <a href=\"enki.html\">Enki</a>\nproject.</span></p>\n<p><span>Thanks to David, Kate, and Raja for\nall their help, and to Jane Street for funding this work.</span></p>\n\n\n\n\n<ol>\n<li><p><span><a href=\"https://github.com/NixOS/nixpkgs/issues/9682\">Which lacks version\nsolving</a>.</span><a href=\"#fnref1\">↩︎</a></p></li>\n</ol>",
9 "content_type": "html",
10 "categories": [],
11 "source": "https://ryan.freumh.org/atom.xml"
12}