Thicket data repository for the EEG
anil.recoil.org
1{
2 "id": "https://www.tunbury.org/2025/05/28/opam2web",
3 "title": "Fix opam2web deployment",
4 "link": "https://www.tunbury.org/2025/05/28/opam2web/",
5 "updated": "2025-05-28T00:00:00",
6 "published": "2025-05-28T00:00:00",
7 "summary": "We maintain a mirror (archive) of all opam packages. To take advantage of this, add the archive mirror to opam by setting the global option.",
8 "content": "<p>We maintain a mirror (archive) of all opam packages. To take advantage of this, add the archive mirror to opam by setting the global option.</p>\n\n<div><div><pre><code>opam option <span>--global</span> <span>'archive-mirrors+=\"https://opam.ocaml.org/cache\"'</span>\n</code></pre></div></div>\n\n<h1>How is the mirror generated and maintained?</h1>\n\n<p>opam has a command that generates the mirror, which defaults to reading <code>packages</code> from the current directory.</p>\n\n<div><div><pre><code>opam admin cache <span>--link</span><span>=</span>archives ./cache\n</code></pre></div></div>\n\n<div>\nsequenceDiagram\n participant BIB as Base Image Builder\n participant DH as Docker Hub\n participant O2W as opam2web\n\n Note over DH: ocaml/opam:archive\n DH-->>BIB: Pull ocaml/opam:archive\n\n Note over BIB: opam admin cache\n BIB->>DH: Push image\n\n Note over DH: ocaml/opam:archive\n DH->>O2W: Pull ocaml/opam:archive\n\n Note over O2W: opam admin cache\n Note over O2W: Publish https://opam.ocaml.org/cache\n</div>\n\n<p>The base image builder pulls <code>ocaml/opam:archive</code>, runs <code>opam admin cache</code> to update the cache, and then pushes it back <code>ocaml/opam:archive</code>.</p>\n\n<p>opam2web, which publishes <a href=\"https://opam.ocaml.org\">opam.ocaml.org</a>, pulls <code>ocaml/opam:archive</code> and then runs <code>opam admin cache</code> to populate any new items in the cache and then makes the cache available at <a href=\"https://opam.ocaml.org/cache\">https://opam.ocaml.org/cache</a>.</p>\n\n<p>Until today, the step indicated by the dotted line was missing. Kate had pointed this out as long ago as 2023 with <a href=\"https://github.com/ocurrent/docker-base-images/issues/249\">issue #249</a> and <a href=\"https://github.com/ocurrent/docker-base-images/pull/248\">PR #248</a>, but, for whatever reason, this was never actioned.</p>\n\n<p>With the current unavailability of <a href=\"https://www.tunbury.org/2025/05/28/opam2web/camlcity.org\">camlcity.org</a>, this has become a problem. On Monday, I patched opam2web’s <code>Dockerfile</code> to include access to the mirror/cache, which allowed opam2web to build. However, subsequent builds failed because the updated <a href=\"https://opam.ocaml.org\">opam.ocaml.org</a> used the latest version of <code>ocaml/opam:archive</code>. This was built on Sunday when camlcity.org was down; therefore, the source for <code>ocamlfind</code> had been dropped from the mirror.</p>\n\n<h1>How to do we get out of this problem?</h1>\n\n<p>Updating the base image builder does not fix the problem, as camlcity.org is still down and the current <code>ocaml/opam:archive</code> does not contain the missing packages. We only tag the latest version on Dockerhub, but looking through the base image builder logs allowed me to find the SHA256 for last week’s build. <code>ocaml/opam:archive@sha256:a0e2cd50e1185fd9a17a193f52d17981a6f9ccf0b56285cbc07f396d5e3f7882</code></p>\n\n<p>Taking <a href=\"https://github.com/ocurrent/docker-base-images/pull/248\">PR #248</a>, and pointing it to the older image, I used the base image builder locally to push an updated <code>ocaml/opam:archive</code>. This is <code>ocaml/opam:archive@sha256:fb7b62ee305b0b9fff82748803e57a655ca92130ab8624476cd7af428101a643</code>.</p>\n\n<div><div><pre><code>- from ~alias:\"opam-archive\" \"ocaml/opam:archive\" @@\n+ from ~alias:\"opam-archive\" \"ocaml/opam:archive@sha256:a0e2cd50e1185fd9a17a193f52d17981a6f9ccf0b56285cbc07f396d5e3f7882\" @@\n</code></pre></div></div>\n\n<p>Now I need to update opam.ocaml.org, but <code>opam2web</code> doesn’t build due to the missing <code>ocamlfind</code>. Checking the <code>opam</code> file showed two source files are needed. One is on GitHub so that’ll be ok.</p>\n\n<div><div><pre><code>...\nurl {\n src: \"http://download.camlcity.org/download/findlib-1.9.6.tar.gz\"\n checksum: [\n \"md5=96c6ee50a32cca9ca277321262dbec57\"\n \"sha512=cfaf1872d6ccda548f07d32cc6b90c3aafe136d2aa6539e03143702171ee0199add55269bba894c77115535dc46a5835901a5d7c75768999e72db503bfd83027\"\n ]\n}\navailable: os != \"win32\"\nextra-source \"0001-Harden-test-for-OCaml-5.patch\" {\n src:\n \"https://raw.githubusercontent.com/ocaml/opam-source-archives/main/patches/ocamlfind/0001-Harden-test-for-OCaml-5.patch\"\n checksum: [\n \"sha256=6fcca5f2f7abf8d6304da6c385348584013ffb8602722a87fb0bacbab5867fe8\"\n \"md5=3cddbf72164c29d4e50e077a92a37c6c\"\n ]\n}\n</code></pre></div></div>\n\n<p>Luck was on my side, as <code>find ~/.opam/download-cache/ -name 96c6ee50a32cca9ca277321262dbec57</code> showed that I had the source in my local opam download cache. I checked out opam2web, copied in the file <code>96c6ee50a32cca9ca277321262dbec57</code> and patched the <code>Dockerfile</code> to inject it into the cache:</p>\n\n<div><div><pre><code>diff --git i/Dockerfile w/Dockerfile\nindex eaf0567..84c9db8 100644\n--- i/Dockerfile\n+++ w/Dockerfile\n@@ -34,6 +34,7 @@ RUN sudo mkdir -p /usr/local/bin \\\n && sudo chmod a+x /usr/local/bin/man2html\n RUN sudo mv /usr/bin/opam-2.3 /usr/bin/opam && opam update\n RUN opam option --global 'archive-mirrors+=\"https://opam.ocaml.org/cache\"'\n+COPY 96c6ee50a32cca9ca277321262dbec57 /home/opam/.opam/download-cache/md5/96/96c6ee50a32cca9ca277321262dbec57\n RUN opam install odoc\n RUN git clone https://github.com/ocaml/opam --single-branch --depth 1 --branch master /home/opam/opam\n WORKDIR /home/opam/opam\n</code></pre></div></div>\n\n<p>The final step is to build and deploy an updated opam2web incorporating the updated mirror cache. In conjunction with the updated base image builder, this will be self-sustaining. I wrapped the necessary steps into a <code>Makefile</code>.</p>\n\n<div><div><pre><code><span>OPAM_REPO_GIT_SHA</span> <span>:=</span> <span>$(</span><span>shell</span> git <span>-C</span> ~/opam-repository fetch upstream <span>&&</span> git <span>-C</span> ~/opam-repository rev-parse upstream/master<span>)</span>\n<span>BLOG_GIT_SHA</span> <span>:=</span> bdef1bbf939db6797dcd51faef2ea9ac1826f4a5\n<span>OPAM_GIT_SHA</span> <span>:=</span> 46234090daf4f9c5f446af56a50f78809c04a20a\n\n<span>all</span><span>:</span> <span>opam2web</span>\n <span>cd</span> <span>opam2web</span> <span>&&</span> <span>docker</span> <span>--context</span> <span>registry.ci.dev</span> <span>build</span> <span>--pull</span> <span>\\</span>\n <span>--build-arg</span> <span>OPAM_REPO_GIT_SHA</span><span>=</span><span>$(OPAM_REPO_GIT_SHA)</span> <span>\\</span>\n <span>--build-arg</span> <span>BLOG_GIT_SHA</span><span>=</span><span>$(BLOG_GIT_SHA)</span> <span>\\</span>\n <span>--build-arg</span> <span>OPAM_GIT_SHA</span><span>=</span><span>$(OPAM_GIT_SHA)</span> <span>\\</span>\n <span>-f</span> Dockerfile <span>--iidfile</span> ../docker-iid <span>--</span> .\n <span>@</span><span>SHA256</span><span>=</span><span>$$</span><span>(</span><span>cat </span>docker-iid<span>)</span>\n <span>docker --context registry.ci.dev tag $$SHA256 registry.ci.dev/opam.ocaml.org</span><span>:</span><span>live</span>\n <span>docker</span> <span>--context</span> <span>registry.ci.dev</span> <span>login</span> <span>-u</span> <span>$(USERNAME)</span> <span>-p</span> <span>$(PASSWORD)</span> <span>registry.ci.dev</span>\n <span>docker --context registry.ci.dev push registry.ci.dev/opam.ocaml.org</span><span>:</span><span>live</span>\n <span>docker --context opam-4.ocaml.org pull registry.ci.dev/opam.ocaml.org</span><span>:</span><span>live</span>\n <span>docker</span> <span>--context</span> <span>opam-4.ocaml.org</span> <span>service</span> <span>update</span> <span>infra_opam_live</span> <span>--image</span> <span>$$SHA256</span>\n <span>docker --context opam-5.ocaml.org pull registry.ci.dev/opam.ocaml.org</span><span>:</span><span>live</span>\n <span>docker</span> <span>--context</span> <span>opam-5.ocaml.org</span> <span>service</span> <span>update</span> <span>infra_opam_live</span> <span>--image</span> <span>$$SHA256</span>\n\n<span>opam2web</span><span>:</span>\n <span>git clone --recursive \"https</span><span>:</span><span>//github.com/ocaml-opam/opam2web.git\" -b \"live\"</span>\n</code></pre></div></div>\n\n<p>Check that <code>ocamlfind</code> is included in the new cache</p>\n\n<div><div><pre><code>wget https://opam-4.ocaml.org/cache/md5/96/96c6ee50a32cca9ca277321262dbec57\nwget https://opam-5.ocaml.org/cache/md5/96/96c6ee50a32cca9ca277321262dbec57\n\n</code></pre></div></div>",
9 "content_type": "html",
10 "author": {
11 "name": "Mark Elvers",
12 "email": "mark.elvers@tunbury.org",
13 "uri": null
14 },
15 "categories": [
16 "opam",
17 "tunbury.org"
18 ],
19 "source": "https://www.tunbury.org/atom.xml"
20}