Thicket data repository for the EEG
anil.recoil.org
1{
2 "id": "https://www.tunbury.org/2025/07/22/package-tool",
3 "title": "Package Tool",
4 "link": "https://www.tunbury.org/2025/07/22/package-tool/",
5 "updated": "2025-07-22T00:00:00",
6 "published": "2025-07-22T00:00:00",
7 "summary": "Would you like to build every package in opam in a single Dockerfile using BuildKit?",
8 "content": "<p>Would you like to build every package in opam in a single Dockerfile using BuildKit?</p>\n\n<p>In <a href=\"https://github.com/mtelvers/package-tool\">mtelvers/package-tool</a>, I have combined various opam sorting and graphing functions into a CLI tool that will work on a checked-out <a href=\"https://github.com/ocaml/opam-repository\">opam-repository</a>. Many of these flags can be combined.</p>\n\n<h1>Package version</h1>\n\n<div><div><pre><code>package-tool <span>--opam-repository</span> ~/opam-repository <package>\n</code></pre></div></div>\n\n<p>The package can be given as <code>0install.2.18</code> or <code>0install</code>. The former specifies a specific version while the latter processes the latest version. <code>--all-versions</code> can be specified to generate files for all package versions.</p>\n\n<h1>Dependencies</h1>\n\n<p>Dump the dependencies for the latest version of 0install into a JSON file.</p>\n\n<div><div><pre><code>package-tool <span>--opam-repository</span> ~/opam-repository <span>--deps</span> 0install\n</code></pre></div></div>\n\n<p>Produces <code>0install.2.18-deps.json</code>:</p>\n\n<div><div><pre><code><span>{</span><span>\"yojson.3.0.0\"</span><span>:[</span><span>\"dune.3.19.1\"</span><span>],</span><span>\n</span><span>\"xmlm.1.4.0\"</span><span>:[</span><span>\"topkg.1.0.8\"</span><span>],</span><span>\n</span><span>\"topkg.1.0.8\"</span><span>:[</span><span>\"ocamlfind.1.9.8\"</span><span>,</span><span>\"ocamlbuild.0.16.1\"</span><span>],</span><span>\n</span><span>...</span><span>\n</span><span>\"0install-solver.2.18\"</span><span>]</span><span>}</span><span>\n</span></code></pre></div></div>\n\n<h1>Installation order</h1>\n\n<p>Create a list showing the installation order for the given package.</p>\n\n<div><div><pre><code>package-tool <span>--opam-repository</span> ~/opam-repository <span>--list</span> 0install\n</code></pre></div></div>\n\n<p>Produces <code>0install.2.18-list.json</code>:</p>\n\n<div><div><pre><code><span>[</span><span>\"ocaml-compiler.5.3.0\"</span><span>,</span><span>\n</span><span>\"ocaml-base-compiler.5.3.0\"</span><span>,</span><span>\n</span><span>...</span><span>\n</span><span>\"0install.2.18\"</span><span>]</span><span>\n</span></code></pre></div></div>\n\n<h1>Solution DAG</h1>\n\n<p>Output the solution graph in Graphviz format, which can then be converted into a PDF with <code>dot</code>.</p>\n\n<div><div><pre><code>package-tool <span>--opam-repository</span> ~/opam-repository <span>--dot</span> 0install\ndot <span>-Tpdf</span> 0install.2.18.dot 0install.2.18.pdf\n</code></pre></div></div>\n<h1>OCaml version</h1>\n\n<p>By default, OCaml 5.3.0 is used, but this can be changed using the <code>--ocaml 4.14.2</code> parameter.</p>\n\n<h1>Dockerfile</h1>\n\n<p>The <code>--dockerfile</code> argument creates a Dockerfile to test the installation.</p>\n\n<div><div><pre><code>package-tool <span>--opam-repository</span> ~/opam-repository <span>--dockerfile</span> <span>--all-versions</span> 0install\n</code></pre></div></div>\n\n<p>For example, the above command line outputs 5 Dockerfiles.</p>\n\n<ul>\n <li>0install.2.15.1.dockerfile</li>\n <li>0install.2.15.2.dockerfile</li>\n <li>0install.2.16.dockerfile</li>\n <li>0install.2.17.dockerfile</li>\n <li>0install.2.18.dockerfile</li>\n</ul>\n\n<p>As an example, <code>0install.2.18.dockerfile</code>, contains:</p>\n\n<div><div><pre><code><span>FROM</span><span> </span><span>debian:12</span><span> </span><span>AS</span><span> </span><span>builder_0install_2_18</span>\n<span>RUN </span>apt update <span>&&</span> apt upgrade <span>-y</span>\n<span>RUN </span>apt <span>install</span> <span>-y</span> build-essential git rsync unzip curl <span>sudo</span>\n<span>RUN if </span>getent passwd 1000<span>;</span> <span>then </span>userdel <span>-r</span> <span>$(</span><span>id</span> <span>-nu</span> 1000<span>)</span><span>;</span> <span>fi</span>\n<span>RUN </span>adduser <span>--uid</span> 1000 <span>--disabled-password</span> <span>--gecos</span> <span>''</span> opam\n<span>ADD</span><span> --chown=root:root --chmod=0755 [ \"https://github.com/ocaml/opam/releases/download/2.3.0/opam-2.3.0-x86_64-linux\", \"/usr/local/bin/opam\" ]</span>\n<span>RUN </span><span>echo</span> <span>'opam ALL=(ALL:ALL) NOPASSWD:ALL'</span> <span>>></span> /etc/sudoers.d/opam\n<span>RUN </span><span>chmod </span>440 /etc/sudoers.d/opam\n<span>USER</span><span> opam</span>\n<span>WORKDIR</span><span> /home/opam</span>\n<span>ENV</span><span> OPAMYES=\"1\" OPAMCONFIRMLEVEL=\"unsafe-yes\" OPAMERRLOGLEN=\"0\" OPAMPRECISETRACKING=\"1\"</span>\n<span>ADD</span><span> --chown=opam:opam --keep-git-dir=false [ \".\", \"/home/opam/opam-repository\" ]</span>\n<span>RUN </span>opam init default <span>-k</span> <span>local</span> ~/opam-repository <span>--disable-sandboxing</span> <span>--bare</span>\n<span>RUN </span>opam switch create default <span>--empty</span>\n<span>RUN </span>opam <span>install </span>ocaml-compiler.5.3.0 <span>>></span> build.log 2>&1 <span>||</span> <span>echo</span> <span>'FAILED'</span> <span>>></span> build.log\n<span>RUN </span>opam <span>install </span>ocaml-base-compiler.5.3.0 <span>>></span> build.log 2>&1 <span>||</span> <span>echo</span> <span>'FAILED'</span> <span>>></span> build.log\n...\n<span>RUN </span>opam <span>install </span>0install-solver.2.18 <span>>></span> build.log 2>&1 <span>||</span> <span>echo</span> <span>'FAILED'</span> <span>>></span> build.log\n<span>RUN </span>opam <span>install </span>0install.2.18 <span>>></span> build.log 2>&1 <span>||</span> <span>echo</span> <span>'FAILED'</span> <span>>></span> build.log\n<span>ENTRYPOINT</span><span> [ \"opam\", \"exec\", \"--\" ]</span>\n<span>CMD</span><span> bash</span>\n</code></pre></div></div>\n\n<p>This can be built using Docker in the normal way. Note that the build context is your checkout of <a href=\"https://github.com/ocaml/opam-repository\">opam-repository</a>.</p>\n\n<div><div><pre><code>docker build <span>-f</span> 0install.2.18.dockerfile ~/opam-repository\n</code></pre></div></div>\n\n<p>Additionally, it outputs <code>Dockerfile</code>, which contains the individual package builds as a multistage build and an aggregation stage as the final layer:</p>\n\n<div><div><pre><code><span>FROM</span><span> </span><span>debian:12</span><span> </span><span>AS</span><span> </span><span>results</span>\n<span>WORKDIR</span><span> /results</span>\n<span>RUN </span>apt update <span>&&</span> apt upgrade <span>-y</span>\n<span>RUN </span>apt <span>install</span> <span>-y</span> less\n<span>COPY</span><span> --from=builder_0install_2_15_1 [ \"/home/opam/build.log\", \"/results/0install.2.15.1\" ]</span>\n<span>COPY</span><span> --from=builder_0install_2_15_2 [ \"/home/opam/build.log\", \"/results/0install.2.15.2\" ]</span>\n<span>COPY</span><span> --from=builder_0install_2_16 [ \"/home/opam/build.log\", \"/results/0install.2.16\" ]</span>\n<span>COPY</span><span> --from=builder_0install_2_17 [ \"/home/opam/build.log\", \"/results/0install.2.17\" ]</span>\n<span>COPY</span><span> --from=builder_0install_2_18 [ \"/home/opam/build.log\", \"/results/0install.2.18\" ]</span>\n<span>CMD</span><span> bash</span>\n</code></pre></div></div>\n\n<p>Build all the versions of 0install in parallel using BuildKit’s layer caching:</p>\n\n<div><div><pre><code>docker build <span>-f</span> Dockerfile <span>-t</span> opam-results ~/opam-repository\n</code></pre></div></div>\n\n<p>We can inspect the build logs in the Docker container:</p>\n\n<div><div><pre><code><span>$ </span>docker run <span>--rm</span> <span>-it</span> opam-results\nroot@b28da667e754:/results# <span>ls</span>^C\nroot@b28da667e754:/results# <span>ls</span> <span>-l</span>\ntotal 76\n<span>-rw-r--r--</span> 1 1000 1000 12055 Jul 22 20:17 0install.2.15.1\n<span>-rw-r--r--</span> 1 1000 1000 15987 Jul 22 20:19 0install.2.15.2\n<span>-rw-r--r--</span> 1 1000 1000 15977 Jul 22 20:19 0install.2.16\n<span>-rw-r--r--</span> 1 1000 1000 16376 Jul 22 20:19 0install.2.17\n<span>-rw-r--r--</span> 1 1000 1000 15150 Jul 22 20:19 0install.2.18\n</code></pre></div></div>\n\n<p>Annoyingly, Docker doesn’t seem to be able to cope with all of opam at once. I get various RPC errors.</p>\n\n<div><div><pre><code>[+] Building 2.9s (4/4) FINISHED docker:default\n => [internal] load build definition from Dockerfile\n => => transferring dockerfile: 10.79MB\n => resolve image config for docker-image://docker.io/docker/dockerfile:1\n => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:9857836c9ee4268391bb5b09f9f157f3c91bb15821bb77969642813b0d00518d\n => [internal] load build definition from Dockerfile\nERROR: failed to receive status: rpc error: code = Unavailable desc = error reading from server: connection error: COMPRESSION_ERROR\n</code></pre></div></div>",
9 "content_type": "html",
10 "author": {
11 "name": "Mark Elvers",
12 "email": "mark.elvers@tunbury.org",
13 "uri": null
14 },
15 "categories": [
16 "OCaml,opam",
17 "tunbury.org"
18 ],
19 "source": "https://www.tunbury.org/atom.xml"
20}