Thicket data repository for the EEG
anil.recoil.org
1{
2 "id": "https://anil.recoil.org/notes/komodo-docker-compose",
3 "title": "Using Komodo to manage Docker compose on a small cluster",
4 "link": "https://anil.recoil.org/notes/komodo-docker-compose",
5 "updated": "2025-05-05T00:00:00",
6 "published": "2025-05-05T00:00:00",
7 "summary": "<p>With the <a href=\"https://www.tunbury.org/equinix-moves/\">sunsetting of Equinix Metal</a>\nI've also been migrating the Recoil machines over to new hosts in <a href=\"https://www.mythic-beasts.com/\">Mythic\nBeasts</a>. This time around, rather than manually\nsetting up services, I've turned to a nice new tool called\n<a href=\"https://github.com/moghtech/komodo\">Komodo</a> which helps with deploying Docker\ncontainers across multiple servers. Unlike many <a href=\"https://kubernetes.io/\">other</a>\ncontainer management solutions, Komodo is refreshingly simple. It has a mode\nwhere it can take <em>existing</em> <a href=\"https://docs.docker.com/compose/\">Docker compose</a> files on a\ngiven host, and run them, and provide a web-based monitor to keep an eye on a\nfew machines.</p>\n<h2><a href=\"https://anil.recoil.org/#the-komodo-interface\"></a>The Komodo interface</h2>\n<p>There's an online <a href=\"https://demo.komo.do/\">demo</a> of Komodo available (user/pass\nis demo/demo). The basic idea is that you first register servers (see below for\n"Periphery"), and then add in "Stacks" which represent a service each.</p>\n<p>\n<img alt=\"The list of Stacks running on Recoil\" src=\"https://anil.recoil.org/images/komodo-1.webp\" title=\"The list of Stacks running on Recoil\">\nThe list of Stacks running on Recoil</p>\n<p>Every stack is configured to run a <code>docker-compose.yml</code> service that is already\npresent on the host, and the web UI has a convenient way of pulling, deploying\nand polling the Docker Hub to check for updates.</p>\n<p>\n<img alt=\"The stack view for a Tangled.sh knot running on Recoil\" src=\"https://anil.recoil.org/images/komodo-2.webp\" title=\"The stack view for a Tangled.sh knot running on Recoil\">\nThe stack view for a Tangled.sh knot running on Recoil</p>\n<p>The autoupdate functionality is quite cool (if a touch risky), as it polls for the\nimages on the Docker Hub and updates to those automagically. While I've activated\nthis for services I'm happy autoupdating, it's also accompanied by a healthy\ndose of <a href=\"https://anil.recoil.org/notes/syncoid-sanoid-zfs\">ZFS snapshotting</a> so I can roll back if anything\nuntoward happens.</p>\n<p>\n<img alt=\"The alert view of autoupdates from polling the Hub\" src=\"https://anil.recoil.org/images/komodo-3.webp\" title=\"The alert view of autoupdates from polling the Hub\">\nThe alert view of autoupdates from polling the Hub</p>\n<p>Most importantly to me is that I can always switch away from Komodo at any time\nand directly interact with the services on the host using the normal <code>docker</code> CLI.\nKomodo is just coordinating the compose invocations in the lightest way possible,\nand not wrapping them in such a way that I lose access.</p>\n<h2><a href=\"https://anil.recoil.org/#setting-up-periphery-with-a-wireguard-mesh-and-dsnet\"></a>Setting up Periphery with a Wireguard mesh and dsnet</h2>\n<p>Komodo operates across multiple hosts by using something called a <a href=\"https://komo.do/docs/connect-servers\">periphery agent</a>\nwhich the main host issues RPCs to in order to do something. This is obviously quite a privileged operation, and so rather than\nexpose it to the Internet I setup a Wireguard tunnel mesh across the Recoil hosts for these operations to go over.</p>\n<p>The easiest way to do this was via <a href=\"https://github.com/naggie/dsnet\">dsnet</a>, which generates the configurations and keys\nsuitable for a <a href=\"https://www.man7.org/linux/man-pages/man8/wg-quick.8.html\">wg-quick</a> service to run on each host and connect\nto their peers. Following the instructions let me setup this mesh in minutes; this is a much simpler solution than\n<a href=\"https://tailscale.com\">Tailscale</a> due to the lack of flexibility, but all I want here is few hosts connected by static interfaces\nand with no need for <a href=\"https://tailscale.com/blog/how-nat-traversal-works\">complex NAT punching</a>. Once the dsnet configuration is\nsetup, all that's needed is to activate the <code>wg-quick</code> service on each of the hosts, and they spin up a virtual interface.</p>\n<p>After this, the Periphery setup was straightforward but with one twist. I configured the agent to bind to the wireguard IP, e.g.:</p>\n<pre><code>/etc/komodo/periphery.config.toml\n################################\n# 🦎 KOMODO PERIPHERY CONFIG 🦎 #\n################################\n\nport = 8120\nbind_ip = "10.100.0.2"\n</code></pre>\n<p>But then on reboot the periphery agent would fail to startup due to the wireguard service being too low a priority in the boot order. This was fixed by a systemd tweak (which took me longer to figure out than the rest of the entire setup altogether, since I find systemd utterly inscrutable).</p>\n<pre><code>/etc/systemd/system/periphery.service\n[Unit]\nDescription=Agent to connect with Komodo Core\nAfter=wg-quick@wg0.service\n</code></pre>\n<p>This little tweak to the script, followed by umpteen <code>daemon-reload</code> prods and\nreboots to get systemd happy, did the trick.</p>\n<p>I'm pretty happy with Komodo, thank you to the devs! It's a system that's simple enough that I can try\nit out progressively, and can bypass easily if required, and provides a very\nuseful part of the <a href=\"https://anil.recoil.org/news?t=selfhosting\">selfhosting</a> jigsaw puzzle.</p>",
8 "content": "<p>With the <a href=\"https://www.tunbury.org/equinix-moves/\">sunsetting of Equinix Metal</a>\nI've also been migrating the Recoil machines over to new hosts in <a href=\"https://www.mythic-beasts.com/\">Mythic\nBeasts</a>. This time around, rather than manually\nsetting up services, I've turned to a nice new tool called\n<a href=\"https://github.com/moghtech/komodo\">Komodo</a> which helps with deploying Docker\ncontainers across multiple servers. Unlike many <a href=\"https://kubernetes.io/\">other</a>\ncontainer management solutions, Komodo is refreshingly simple. It has a mode\nwhere it can take <em>existing</em> <a href=\"https://docs.docker.com/compose/\">Docker compose</a> files on a\ngiven host, and run them, and provide a web-based monitor to keep an eye on a\nfew machines.</p>\n<h2><a href=\"https://anil.recoil.org/#the-komodo-interface\"></a>The Komodo interface</h2>\n<p>There's an online <a href=\"https://demo.komo.do/\">demo</a> of Komodo available (user/pass\nis demo/demo). The basic idea is that you first register servers (see below for\n"Periphery"), and then add in "Stacks" which represent a service each.</p>\n<p>\n<img alt=\"The list of Stacks running on Recoil\" src=\"https://anil.recoil.org/images/komodo-1.webp\" title=\"The list of Stacks running on Recoil\">\nThe list of Stacks running on Recoil</p>\n<p>Every stack is configured to run a <code>docker-compose.yml</code> service that is already\npresent on the host, and the web UI has a convenient way of pulling, deploying\nand polling the Docker Hub to check for updates.</p>\n<p>\n<img alt=\"The stack view for a Tangled.sh knot running on Recoil\" src=\"https://anil.recoil.org/images/komodo-2.webp\" title=\"The stack view for a Tangled.sh knot running on Recoil\">\nThe stack view for a Tangled.sh knot running on Recoil</p>\n<p>The autoupdate functionality is quite cool (if a touch risky), as it polls for the\nimages on the Docker Hub and updates to those automagically. While I've activated\nthis for services I'm happy autoupdating, it's also accompanied by a healthy\ndose of <a href=\"https://anil.recoil.org/notes/syncoid-sanoid-zfs\">ZFS snapshotting</a> so I can roll back if anything\nuntoward happens.</p>\n<p>\n<img alt=\"The alert view of autoupdates from polling the Hub\" src=\"https://anil.recoil.org/images/komodo-3.webp\" title=\"The alert view of autoupdates from polling the Hub\">\nThe alert view of autoupdates from polling the Hub</p>\n<p>Most importantly to me is that I can always switch away from Komodo at any time\nand directly interact with the services on the host using the normal <code>docker</code> CLI.\nKomodo is just coordinating the compose invocations in the lightest way possible,\nand not wrapping them in such a way that I lose access.</p>\n<h2><a href=\"https://anil.recoil.org/#setting-up-periphery-with-a-wireguard-mesh-and-dsnet\"></a>Setting up Periphery with a Wireguard mesh and dsnet</h2>\n<p>Komodo operates across multiple hosts by using something called a <a href=\"https://komo.do/docs/connect-servers\">periphery agent</a>\nwhich the main host issues RPCs to in order to do something. This is obviously quite a privileged operation, and so rather than\nexpose it to the Internet I setup a Wireguard tunnel mesh across the Recoil hosts for these operations to go over.</p>\n<p>The easiest way to do this was via <a href=\"https://github.com/naggie/dsnet\">dsnet</a>, which generates the configurations and keys\nsuitable for a <a href=\"https://www.man7.org/linux/man-pages/man8/wg-quick.8.html\">wg-quick</a> service to run on each host and connect\nto their peers. Following the instructions let me setup this mesh in minutes; this is a much simpler solution than\n<a href=\"https://tailscale.com\">Tailscale</a> due to the lack of flexibility, but all I want here is few hosts connected by static interfaces\nand with no need for <a href=\"https://tailscale.com/blog/how-nat-traversal-works\">complex NAT punching</a>. Once the dsnet configuration is\nsetup, all that's needed is to activate the <code>wg-quick</code> service on each of the hosts, and they spin up a virtual interface.</p>\n<p>After this, the Periphery setup was straightforward but with one twist. I configured the agent to bind to the wireguard IP, e.g.:</p>\n<pre><code>/etc/komodo/periphery.config.toml\n################################\n# 🦎 KOMODO PERIPHERY CONFIG 🦎 #\n################################\n\nport = 8120\nbind_ip = "10.100.0.2"\n</code></pre>\n<p>But then on reboot the periphery agent would fail to startup due to the wireguard service being too low a priority in the boot order. This was fixed by a systemd tweak (which took me longer to figure out than the rest of the entire setup altogether, since I find systemd utterly inscrutable).</p>\n<pre><code>/etc/systemd/system/periphery.service\n[Unit]\nDescription=Agent to connect with Komodo Core\nAfter=wg-quick@wg0.service\n</code></pre>\n<p>This little tweak to the script, followed by umpteen <code>daemon-reload</code> prods and\nreboots to get systemd happy, did the trick.</p>\n<p>I'm pretty happy with Komodo, thank you to the devs! It's a system that's simple enough that I can try\nit out progressively, and can bypass easily if required, and provides a very\nuseful part of the <a href=\"https://anil.recoil.org/news?t=selfhosting\">selfhosting</a> jigsaw puzzle.</p>",
9 "content_type": "html",
10 "author": {
11 "name": "Anil Madhavapeddy",
12 "email": "anil@recoil.org",
13 "uri": "https://anil.recoil.org"
14 },
15 "categories": [],
16 "rights": "(c) 1998-2025 Anil Madhavapeddy, all rights reserved",
17 "source": "https://anil.recoil.org/news.xml"
18}