Thicket data repository for the EEG
anil.recoil.org
1{
2 "id": "https://ryan.freumh.org/2024-03-25.html",
3 "title": "25 Mar 2024",
4 "link": "https://ryan.freumh.org/2024-03-25.html",
5 "updated": "2024-03-25T00:00:00",
6 "published": "2024-03-25T00:00:00",
7 "summary": "<div>\n <span> Previous: <a href=\"2024-03-18.html\">18 Mar 2024</a> </span>\n <span> Next: <a href=\"2024-04-01.html\"> 1 Apr 2024</a> </span>\n </div>\n \n \n\n <h2>Writing</h2>\n<p><span>Scientific method: My thesis is that the\nDNS is the root of the Internet. That is, every network request starts\noff with a DNS resolution. The DNS is in a great position to manage\nresources in the Network, like certificates, keys, services, etc. But\nfor some reason people don’t do this now.</span></p>\n<p><span>One advantage Eon has over the\nstate-of-the-art dynamic DNS updates for the ACME DNS challenge is the\nprinciple of least privilege. We can provide a capability to only a\nsubdomain, with the thread model a rough service or machine.</span></p>\n<p><span>Regarding scenario 5, the DNS is a very\nsmall key value store. “The DNS is a general (if somewhat limited)\nhierarchical database, and can store almost any kind of data, for almost\nany purpose,” RFC2181. We can keep track of what values are being used.\nLike GitHub s SSH keys, and Tailscale’s key expiry. These could be SSH\nkeys for machines with SSHFP. DNSSEC would be nice for this. A UI into\nthis might be appropriate, possibly authenticated with something like <a href=\"https://webauthn.io/\">webauthn.io</a></span></p>\n<p><span>Additional scenarios:</span></p>\n<ol>\n<li>Self-healing OS.</li>\n</ol>\n<p><span>Setting up and configuring machines is a\ntime and labour-intensive process. Installing the operating system,\nnetworking the machine, updating software, never mind deploying\nservices. Instead, we could deploy a machine just with a capability to\naccess a name service, which could register itself and boot whatever\nservices are configured. It could manage its own upgrades, resources\nincluding disk space, etc.</span></p>\n<ol>\n<li>Bridging domains.</li>\n</ol>\n<p><span>TBD…</span></p>\n<h2>Reading</h2>\n<p><span>I was looking at the godot engine as an\nopen source alternative to unity for mixed reality development and came\nacross <a href=\"https://github.com/Nitwel/Immersive-Home\">github.com/Nitwel/Immersive-Home</a>.</span></p>\n<p><span>On the topic of a self-healing OS, one\ngripe I’ve had with NixOS since I started using it is that it doesn’t\nsupport mutable runtime state. It turns out there’s a masters thesis\nexploring this exact topic <a href=\"http://denbreejen.net/web/thesis.pdf\">denbreejen.net/web/thesis.pdf</a>.\nSome of these ideas could be useful for a self-healing OS. I could\nimagine a modern solution using container technologies like Linux\nnamespaces to achieve this.</span></p>\n<h3><a href=\"http://denbreejen.net/web/thesis.pdf\">Managing state in a purely\nfunctional deployment model</a></h3>\n<p><span>The\npurely functional deployment model doesn’t extend to mutable system\nstate such as databases and log files. Managing state out-of-band\nhinders the reproducibility of deployments. For example, when migrating\na component users are required to migrate the state separately, there’s\nno mechanism to specify multiple instances of a component that use\ndifferent instances of state, rolling back software versions may be\nincompatible with state, and similarly upgrading software often requires\na state transformation. The versioning file system Ext3cow is used to\nstore multiple instances of state. This seems similar to ZFS\nsnapshotting.</span></p>\n<p><span>They\ntreat state as analogous to values in a purely functional language.\nExecutions of components are transformations of state</span></p>\n<pre><code>execute_component :: Component -> State -> State</code></pre>\n<p><span>State\nis stored in a state store <code>/nix/state</code>. This is extending\nNix to be more active in the runtime.</span></p>\n<p><span>As\nfar as I can tell this hasn’t seen real-world adoption. Perhaps as it\nconstrains the deployment unduly (requiring specific file\nsystem).</span></p>\n<p><span><a href=\"https://github.com/svanderburg/disnix\">disnix</a></span></p>\n<h2>Hacking</h2>\n<p><span>I need to modify the NixOS ACME config that\nuses the lego bot to instead use Eon to provision certificates for my\nown VPS as well as Eon.</span></p>\n<p><span>The way the <a href=\"https://github.com/NixOS/nixpkgs/blob/56528ee42526794d413d6f244648aaee4a7b56c0/nixos/modules/services/web-servers/nginx/default.nix#L1215\">existing\nconfiguration</a> works is that a systemd services is created for every\ncertificate to handle renewals, and <a href=\"https://github.com/NixOS/nixpkgs/blob/56528ee42526794d413d6f244648aaee4a7b56c0/nixos/modules/services/web-servers/nginx/default.nix#L1215\">webserver\nservices</a> are set up to restart on a new certificate being\nprovisioned.</span></p>\n<p><span>The simplest solution seems like modifying\nthe systemd service to invoke a small client to provision a certificate.\nWe can get the capability for a domain from a file protected with file\npermissions (e.g. allow the nginx group access). This is similar access\ncontrol to a Unix domain socket, but we’re separating access control and\nnetwork connectivity, so a service could be running on another machine\nand still use this capability.</span></p>\n<p><span>We’re inverting the renewal logic here.\nCurrently, systemd triggers timers for renewal scripts, but Eon can\nhandle certificate renewals with callbacks. We’ll need a mechanism to\nrestart the webserver when certificates are renewed. We could probably\naccomplish this for systemd with <a href=\"https://www.freedesktop.org/software/systemd/man/latest/systemd.path.html\">paths</a>\nto the cert files. Or just having the client exit on renewal, triggering\na restart of the systemd unit, and restarting the client with the same\nrenewal callback.</span></p>\n<p><span>We’re only talking about restarting the\nwebserver as it looks like every HTTP service (matrix, mastodon, etc) is\nexpected to run behind a reverse proxy which handles things like TLS.\n(With SVCB records specifying a port, maybe this is\nunnecessary.)</span></p>\n<p><span>I don’t think this is not fulfilling the\nfull capability of this approach, but is a worthwhile MVP to use it in\ndeployment. For dynamically created services, a small wrapper executable\nto provision certs and restart on renewal seems appropriate. This is\nmore like docker compose.</span></p>\n<p><span>I’m in the middle of hacking in Nix to get\nthis working.</span></p>\n<h2>polyglot package management</h2>\n<p><span>David was very keen on\nthe project proposal as it mirrored a lot of what he’d been thinking\nabout from the Opam for Windows perspective. The reasons why he thinks\nNix isn’t sufficient by itself are:</span></p>\n<ul>\n<li>Platform support</li>\n<li>Version solving</li>\n<li>A lack of ‘just works’ tooling</li>\n<li>He’s suspicious of monolithic build system approaches to package\nmanagement (e.g. bazel) as they just recreate the world in their own\nformat, and as package management is orthogonal to build systems.</li>\n</ul>\n<p><span>We talked a lot about\nhis own ideas for Opam (a lot of it in the context of windows). He\ntalked about dynamically generating opam files from what is available\nvia the system package manager, and feeding that back to the solver.\nIt’s really useful stuff to know, but he said it shouldn’t affect the\ndeliverables for this proposal.</span></p>\n<p><span>We talked about the\nlimitations of trying to build a universal package manager to invoke\nothers, and how a common format could work (though this ‘holy grail’ is\na bit of a pipe dream at the moment). We also talked about Dune’s\nupcoming package management functionality.</span></p>\n<p><span>He’s really keen on the\ndream2nix proposal 2 for supporting the Opam solver. But the thought\nproposal 1 a Nix backend for Opam depexts was the lower hanging fruit to\nbe getting on with in the first instance. This is essentially indexing\nNixpkgs for versions of packages, and solving for the correct version\nwith the opam solver.</span></p>\n<p><span>He said that the best\nway to support this might be an Opam plugin. Though it might require\nchanges to the depext logic e.g. for sandboxing and environement\nvariables. The depopts field of the opam file might be the place to\nspecify this, as conf packages don’t really support versions like we\ndo.</span></p>\n<p><span>The\n<code>nix-shell -p <package></code> command makes available\nbinaries in the $PATH variable. I need to think about the best way to\nmake this available to the Opam sandbox.</span></p>\n<p><span>David said setenv might\nbe what we need. The command also makes available libraries in various\nlanguage specific environment variables, like $NIX_LDFLAGS (for the C\ncompiler wrapper) and $PYTHONPATH for Python. I need to think about if\nand how these could be supported for shared libraries.</span></p>\n<p><span>The next task is\nprobably modifying <a href=\"https://github.com/lazamar/nix-package-versions\">https://github.com/lazamar/nix-package-versions</a>\nto index Nixpkgs in our Opam repo.</span></p>",
8 "content": "<div>\n <span> Previous: <a href=\"2024-03-18.html\">18 Mar 2024</a> </span>\n <span> Next: <a href=\"2024-04-01.html\"> 1 Apr 2024</a> </span>\n </div>\n \n \n\n <h2>Writing</h2>\n<p><span>Scientific method: My thesis is that the\nDNS is the root of the Internet. That is, every network request starts\noff with a DNS resolution. The DNS is in a great position to manage\nresources in the Network, like certificates, keys, services, etc. But\nfor some reason people don’t do this now.</span></p>\n<p><span>One advantage Eon has over the\nstate-of-the-art dynamic DNS updates for the ACME DNS challenge is the\nprinciple of least privilege. We can provide a capability to only a\nsubdomain, with the thread model a rough service or machine.</span></p>\n<p><span>Regarding scenario 5, the DNS is a very\nsmall key value store. “The DNS is a general (if somewhat limited)\nhierarchical database, and can store almost any kind of data, for almost\nany purpose,” RFC2181. We can keep track of what values are being used.\nLike GitHub s SSH keys, and Tailscale’s key expiry. These could be SSH\nkeys for machines with SSHFP. DNSSEC would be nice for this. A UI into\nthis might be appropriate, possibly authenticated with something like <a href=\"https://webauthn.io/\">webauthn.io</a></span></p>\n<p><span>Additional scenarios:</span></p>\n<ol>\n<li>Self-healing OS.</li>\n</ol>\n<p><span>Setting up and configuring machines is a\ntime and labour-intensive process. Installing the operating system,\nnetworking the machine, updating software, never mind deploying\nservices. Instead, we could deploy a machine just with a capability to\naccess a name service, which could register itself and boot whatever\nservices are configured. It could manage its own upgrades, resources\nincluding disk space, etc.</span></p>\n<ol>\n<li>Bridging domains.</li>\n</ol>\n<p><span>TBD…</span></p>\n<h2>Reading</h2>\n<p><span>I was looking at the godot engine as an\nopen source alternative to unity for mixed reality development and came\nacross <a href=\"https://github.com/Nitwel/Immersive-Home\">github.com/Nitwel/Immersive-Home</a>.</span></p>\n<p><span>On the topic of a self-healing OS, one\ngripe I’ve had with NixOS since I started using it is that it doesn’t\nsupport mutable runtime state. It turns out there’s a masters thesis\nexploring this exact topic <a href=\"http://denbreejen.net/web/thesis.pdf\">denbreejen.net/web/thesis.pdf</a>.\nSome of these ideas could be useful for a self-healing OS. I could\nimagine a modern solution using container technologies like Linux\nnamespaces to achieve this.</span></p>\n<h3><a href=\"http://denbreejen.net/web/thesis.pdf\">Managing state in a purely\nfunctional deployment model</a></h3>\n<p><span>The\npurely functional deployment model doesn’t extend to mutable system\nstate such as databases and log files. Managing state out-of-band\nhinders the reproducibility of deployments. For example, when migrating\na component users are required to migrate the state separately, there’s\nno mechanism to specify multiple instances of a component that use\ndifferent instances of state, rolling back software versions may be\nincompatible with state, and similarly upgrading software often requires\na state transformation. The versioning file system Ext3cow is used to\nstore multiple instances of state. This seems similar to ZFS\nsnapshotting.</span></p>\n<p><span>They\ntreat state as analogous to values in a purely functional language.\nExecutions of components are transformations of state</span></p>\n<pre><code>execute_component :: Component -> State -> State</code></pre>\n<p><span>State\nis stored in a state store <code>/nix/state</code>. This is extending\nNix to be more active in the runtime.</span></p>\n<p><span>As\nfar as I can tell this hasn’t seen real-world adoption. Perhaps as it\nconstrains the deployment unduly (requiring specific file\nsystem).</span></p>\n<p><span><a href=\"https://github.com/svanderburg/disnix\">disnix</a></span></p>\n<h2>Hacking</h2>\n<p><span>I need to modify the NixOS ACME config that\nuses the lego bot to instead use Eon to provision certificates for my\nown VPS as well as Eon.</span></p>\n<p><span>The way the <a href=\"https://github.com/NixOS/nixpkgs/blob/56528ee42526794d413d6f244648aaee4a7b56c0/nixos/modules/services/web-servers/nginx/default.nix#L1215\">existing\nconfiguration</a> works is that a systemd services is created for every\ncertificate to handle renewals, and <a href=\"https://github.com/NixOS/nixpkgs/blob/56528ee42526794d413d6f244648aaee4a7b56c0/nixos/modules/services/web-servers/nginx/default.nix#L1215\">webserver\nservices</a> are set up to restart on a new certificate being\nprovisioned.</span></p>\n<p><span>The simplest solution seems like modifying\nthe systemd service to invoke a small client to provision a certificate.\nWe can get the capability for a domain from a file protected with file\npermissions (e.g. allow the nginx group access). This is similar access\ncontrol to a Unix domain socket, but we’re separating access control and\nnetwork connectivity, so a service could be running on another machine\nand still use this capability.</span></p>\n<p><span>We’re inverting the renewal logic here.\nCurrently, systemd triggers timers for renewal scripts, but Eon can\nhandle certificate renewals with callbacks. We’ll need a mechanism to\nrestart the webserver when certificates are renewed. We could probably\naccomplish this for systemd with <a href=\"https://www.freedesktop.org/software/systemd/man/latest/systemd.path.html\">paths</a>\nto the cert files. Or just having the client exit on renewal, triggering\na restart of the systemd unit, and restarting the client with the same\nrenewal callback.</span></p>\n<p><span>We’re only talking about restarting the\nwebserver as it looks like every HTTP service (matrix, mastodon, etc) is\nexpected to run behind a reverse proxy which handles things like TLS.\n(With SVCB records specifying a port, maybe this is\nunnecessary.)</span></p>\n<p><span>I don’t think this is not fulfilling the\nfull capability of this approach, but is a worthwhile MVP to use it in\ndeployment. For dynamically created services, a small wrapper executable\nto provision certs and restart on renewal seems appropriate. This is\nmore like docker compose.</span></p>\n<p><span>I’m in the middle of hacking in Nix to get\nthis working.</span></p>\n<h2>polyglot package management</h2>\n<p><span>David was very keen on\nthe project proposal as it mirrored a lot of what he’d been thinking\nabout from the Opam for Windows perspective. The reasons why he thinks\nNix isn’t sufficient by itself are:</span></p>\n<ul>\n<li>Platform support</li>\n<li>Version solving</li>\n<li>A lack of ‘just works’ tooling</li>\n<li>He’s suspicious of monolithic build system approaches to package\nmanagement (e.g. bazel) as they just recreate the world in their own\nformat, and as package management is orthogonal to build systems.</li>\n</ul>\n<p><span>We talked a lot about\nhis own ideas for Opam (a lot of it in the context of windows). He\ntalked about dynamically generating opam files from what is available\nvia the system package manager, and feeding that back to the solver.\nIt’s really useful stuff to know, but he said it shouldn’t affect the\ndeliverables for this proposal.</span></p>\n<p><span>We talked about the\nlimitations of trying to build a universal package manager to invoke\nothers, and how a common format could work (though this ‘holy grail’ is\na bit of a pipe dream at the moment). We also talked about Dune’s\nupcoming package management functionality.</span></p>\n<p><span>He’s really keen on the\ndream2nix proposal 2 for supporting the Opam solver. But the thought\nproposal 1 a Nix backend for Opam depexts was the lower hanging fruit to\nbe getting on with in the first instance. This is essentially indexing\nNixpkgs for versions of packages, and solving for the correct version\nwith the opam solver.</span></p>\n<p><span>He said that the best\nway to support this might be an Opam plugin. Though it might require\nchanges to the depext logic e.g. for sandboxing and environement\nvariables. The depopts field of the opam file might be the place to\nspecify this, as conf packages don’t really support versions like we\ndo.</span></p>\n<p><span>The\n<code>nix-shell -p <package></code> command makes available\nbinaries in the $PATH variable. I need to think about the best way to\nmake this available to the Opam sandbox.</span></p>\n<p><span>David said setenv might\nbe what we need. The command also makes available libraries in various\nlanguage specific environment variables, like $NIX_LDFLAGS (for the C\ncompiler wrapper) and $PYTHONPATH for Python. I need to think about if\nand how these could be supported for shared libraries.</span></p>\n<p><span>The next task is\nprobably modifying <a href=\"https://github.com/lazamar/nix-package-versions\">https://github.com/lazamar/nix-package-versions</a>\nto index Nixpkgs in our Opam repo.</span></p>",
9 "content_type": "html",
10 "categories": [],
11 "source": "https://ryan.freumh.org/atom.xml"
12}