mte/2025_06_27_windows-containerd-3.json at main · anil.recoil.org/thicket-eeg

Thicket data repository for the EEG
thicket-eeg / mte / 2025_06_27_windows-containerd-3.json
at main 19 kB view raw
 1{
 2  "id": "https://www.tunbury.org/2025/06/27/windows-containerd-3",
 3  "title": "Containerd on Windows",
 4  "link": "https://www.tunbury.org/2025/06/27/windows-containerd-3/",
 5  "updated": "2025-06-27T12:00:00",
 6  "published": "2025-06-27T12:00:00",
 7  "summary": "Everything was going fine until I ran out of disk space. My NVMe, C: drive, is only 256GB, but I have a large, 1.7TB SSD available as D:. How trivial, change a few paths and carry on, but it wasn’t that simple, or was it?",
 8  "content": "<p>Everything was going fine until I ran out of disk space. My NVMe, <code>C:</code> drive, is only 256GB, but I have a large, 1.7TB SSD available as <code>D:</code>. How trivial, change a few paths and carry on, but it wasn’t that simple, or was it?</p>\n\n<p>Distilling the problem down to the minimum and excluding all code written by me, the following command fails, but changing <code>src=d:\\cache\\opam</code> to <code>src=c:\\cache\\opam</code> works. It’s not the content, as it’s just an empty folder.</p>\n\n<pre><code>ctr run --rm --cni -user ContainerAdministrator -mount type=bind,src=d:\\cache\\opam,dst=c:\\Users\\ContainerAdministrator\\AppData\\Local\\opam mcr.microsoft.com/windows/servercore:ltsc2022 my-container  cmd /c \"curl.exe -L -o c:\\Windows\\opam.exe https://github.com/ocaml/opam/releases/download/2.3.0/opam-2.3.0-x86_64-windows.exe &amp;&amp; opam.exe init --debug-level=3 -y\"\n</code></pre>\n\n<p>The failure point is the ability to create the lock file <code>config.lock</code>. Checking the code, the log entry is written before the lock is acquired. If <code>c:\\Users\\ContainerAdministrator\\AppData\\Local\\opam</code> is not a bind mount, or the bind mount is on <code>C:</code>, then it works.</p>\n\n<div><div><pre><code>01:26.722  CLIENT                          updating repository state\n01:26.722  GSTATE                          LOAD-GLOBAL-STATE @ C:\\Users\\ContainerAdministrator\\AppData\\Local\\opam\n01:26.723  SYSTEM                          LOCK C:\\Users\\ContainerAdministrator\\AppData\\Local\\opam\\lock (none =&gt; read)\n01:26.723  SYSTEM                          LOCK C:\\Users\\ContainerAdministrator\\AppData\\Local\\opam\\config.lock (none =&gt; write)\n</code></pre></div></div>\n\n<p>Suffice it to say, I spent a long time trying to resolve this. I’ll mention a couple of interesting points that appeared along the way. Firstly, files created on <code>D:</code> effectively appear as hard links, and the Update Sequence Number, USN, is 0.</p>\n\n<div><div><pre><code><span>C:\\</span><span>&gt;</span><span> </span><span>fsutil</span><span> </span><span>file</span><span> </span><span>layout</span><span> </span><span>d:\\cache\\opam\\lock</span><span>\n\n</span><span>*********</span><span> </span><span>File</span><span> </span><span>0x000400000001d251</span><span> </span><span>*********</span><span>\n</span><span>File</span><span> </span><span>reference</span><span> </span><span>number</span><span>   </span><span>:</span><span> </span><span>0x000400000001d251</span><span>\n</span><span>File</span><span> </span><span>attributes</span><span>         </span><span>:</span><span> </span><span>0x00000020:</span><span> </span><span>Archive</span><span>\n</span><span>File</span><span> </span><span>entry</span><span> </span><span>flags</span><span>        </span><span>:</span><span> </span><span>0x00000000</span><span>\n</span><span>Link</span><span> </span><span>(</span><span>ParentID:</span><span> </span><span>Name</span><span>)</span><span>   </span><span>:</span><span> </span><span>0</span><span>x000c00000000002d:</span><span> </span><span>HLINK</span><span> </span><span>Name</span><span>   </span><span>:</span><span> </span><span>\\cache\\opam\\lock</span><span>\n</span><span>...</span><span>\n</span><span>LastUsn</span><span>                 </span><span>:</span><span> </span><span>0</span><span>\n</span><span>...</span><span>\n</span></code></pre></div></div>\n\n<p>The reason behind this is down to Windows defaults:</p>\n\n<ol>\n  <li>Windows still likes to create the legacy 8.3 MS-DOS file names on the system volume, <code>C:</code>, which explains the difference between <code>HLINK</code> and <code>NTFS+DOS</code>. Running <code>fsutil 8dot3name set d: 0</code> will enable the creation of the old-style file names.</li>\n  <li>Drive <code>C:</code> has a USN journal created automatically, as it’s required for Windows to operate, but it isn’t created by default on other drives. Running <code>fsutil usn createjournal d: m=32000000 a=8000000</code> will create the journal.</li>\n</ol>\n\n<div><div><pre><code><span>C:\\</span><span>&gt;</span><span> </span><span>fsutil</span><span> </span><span>file</span><span> </span><span>layout</span><span> </span><span>c:\\cache\\opam\\lock</span><span>\n\n</span><span>*********</span><span> </span><span>File</span><span> </span><span>0x000300000002f382</span><span> </span><span>*********</span><span>\n</span><span>File</span><span> </span><span>reference</span><span> </span><span>number</span><span>   </span><span>:</span><span> </span><span>0x000300000002f382</span><span>\n</span><span>File</span><span> </span><span>attributes</span><span>         </span><span>:</span><span> </span><span>0x00000020:</span><span> </span><span>Archive</span><span>\n</span><span>File</span><span> </span><span>entry</span><span> </span><span>flags</span><span>        </span><span>:</span><span> </span><span>0x00000000</span><span>\n</span><span>Link</span><span> </span><span>(</span><span>ParentID:</span><span> </span><span>Name</span><span>)</span><span>   </span><span>:</span><span> </span><span>0</span><span>x000b0000000271d1:</span><span> </span><span>NTFS</span><span>+</span><span>DOS</span><span> </span><span>Name:</span><span> </span><span>\\cache\\opam\\lock</span><span>\n</span><span>...</span><span>\n</span><span>LastUsn</span><span>                 </span><span>:</span><span> </span><span>16</span><span>,</span><span>897</span><span>,</span><span>595</span><span>,</span><span>224</span><span>\n</span><span>...</span><span>\n</span></code></pre></div></div>\n\n<p>Sadly, neither of these insights makes any difference to my problem. I did notice that <code>containerd</code> 2.1.3 had been released, where I had been using 2.1.1. Upgrading didn’t fix the issue, but it did affect how the network namespaces were created. More later.</p>\n\n<p>I decided to both ignore the problem and try it on another machine. After all, this problem was only a problem because <em>my</em> <code>C:</code> was too small. I created a QEMU VM with a 40GB <code>C:</code> and a 1TB <code>D:</code> and installed everything, and it worked fine with the bind mount on <code>D:</code> even <em>without</em> any of the above tuning and even with <code>D:</code> formatted using ReFS, rather than NTFS.</p>\n\n<p>Trying on another physical machine with a single large spinning disk as <code>C:</code> also worked as anticipated.</p>\n\n<p>In both of these new installations, I used <code>containerd</code> 2.1.3 and noticed that the behaviour I had come to rely upon seemed to have changed. If you recall, in this <a href=\"https://www.tunbury.org/2025/06/14/windows-containerd-2/\">post</a>, I <em>found</em> the network namespace GUID by running <code>ctr run</code> on a standard Windows container and then <code>ctr container info</code> in another window. This no longer worked reliably, as the namespace was removed when the container exited. Perhaps it always should have been?</p>\n\n<p>I need to find out how to create these namespaces. PowerShell has a cmdlet <code>Get-HnsNetwork</code>, but none of the GUID values there match the currently running namespaces I observe from <code>ctr container info</code>. The source code of <a href=\"https://github.com/containerd/containerd\">containerd</a> is on GitHub..</p>\n\n<p>When you pass <code>--cni</code> to the <code>ctr</code> command, it populates the network namespace from <code>NetNewNS</code>.  Snippet from <code>cmd/ctr/commands/run/run_windows.go</code></p>\n\n<div><div><pre><code>                <span>if</span> <span>cliContext</span><span>.</span><span>Bool</span><span>(</span><span>\"cni\"</span><span>)</span> <span>{</span>\n                        <span>ns</span><span>,</span> <span>err</span> <span>:=</span> <span>netns</span><span>.</span><span>NewNetNS</span><span>(</span><span>\"\"</span><span>)</span>\n                        <span>if</span> <span>err</span> <span>!=</span> <span>nil</span> <span>{</span>\n                                <span>return</span> <span>nil</span><span>,</span> <span>err</span>\n                        <span>}</span>\n                        <span>opts</span> <span>=</span> <span>append</span><span>(</span><span>opts</span><span>,</span> <span>oci</span><span>.</span><span>WithWindowsNetworkNamespace</span><span>(</span><span>ns</span><span>.</span><span>GetPath</span><span>()))</span>\n                <span>}</span>\n</code></pre></div></div>\n\n<p><code>NewNetNS</code> is defined in <code>pkg/netns/netns_windows.go</code></p>\n\n<div><div><pre><code><span>// NetNS holds network namespace for sandbox</span>\n<span>type</span> <span>NetNS</span> <span>struct</span> <span>{</span>\n        <span>path</span> <span>string</span>\n<span>}</span>\n\n<span>// NewNetNS creates a network namespace for the sandbox.</span>\n<span>func</span> <span>NewNetNS</span><span>(</span><span>baseDir</span> <span>string</span><span>)</span> <span>(</span><span>*</span><span>NetNS</span><span>,</span> <span>error</span><span>)</span> <span>{</span>\n        <span>temp</span> <span>:=</span> <span>hcn</span><span>.</span><span>HostComputeNamespace</span><span>{}</span>\n        <span>hcnNamespace</span><span>,</span> <span>err</span> <span>:=</span> <span>temp</span><span>.</span><span>Create</span><span>()</span>\n        <span>if</span> <span>err</span> <span>!=</span> <span>nil</span> <span>{</span>\n                <span>return</span> <span>nil</span><span>,</span> <span>err</span>\n        <span>}</span>\n\n        <span>return</span> <span>&amp;</span><span>NetNS</span><span>{</span><span>path</span><span>:</span> <span>hcnNamespace</span><span>.</span><span>Id</span><span>},</span> <span>nil</span>\n<span>}</span>\n</code></pre></div></div>\n\n<p>Following the thread, and cutting out a few steps in the interest of brevity, we end up in <code>vendor/github.com/Microsoft/hcsshim/hcn/zsyscall_windows.go</code> which calls a Win32 API.</p>\n\n<div><div><pre><code><span>func</span> <span>_hcnCreateNamespace</span><span>(</span><span>id</span> <span>*</span><span>_guid</span><span>,</span> <span>settings</span> <span>*</span><span>uint16</span><span>,</span> <span>namespace</span> <span>*</span><span>hcnNamespace</span><span>,</span> <span>result</span> <span>**</span><span>uint16</span><span>)</span> <span>(</span><span>hr</span> <span>error</span><span>)</span> <span>{</span>\n        <span>hr</span> <span>=</span> <span>procHcnCreateNamespace</span><span>.</span><span>Find</span><span>()</span>\n        <span>if</span> <span>hr</span> <span>!=</span> <span>nil</span> <span>{</span>\n                <span>return</span>\n        <span>}</span>\n        <span>r0</span><span>,</span> <span>_</span><span>,</span> <span>_</span> <span>:=</span> <span>syscall</span><span>.</span><span>SyscallN</span><span>(</span><span>procHcnCreateNamespace</span><span>.</span><span>Addr</span><span>(),</span> <span>uintptr</span><span>(</span><span>unsafe</span><span>.</span><span>Pointer</span><span>(</span><span>id</span><span>)),</span> <span>uintptr</span><span>(</span><span>unsafe</span><span>.</span><span>Pointer</span><span>(</span><span>settings</span><span>)),</span> <span>uintptr</span><span>(</span><span>unsafe</span><span>.</span><span>Pointer</span><span>(</span><span>namespace</span><span>)),</span> <span>uintptr</span><span>(</span><span>unsafe</span><span>.</span><span>Pointer</span><span>(</span><span>result</span><span>)))</span>\n        <span>if</span> <span>int32</span><span>(</span><span>r0</span><span>)</span> <span>&lt;</span> <span>0</span> <span>{</span>\n                <span>if</span> <span>r0</span><span>&amp;</span><span>0x1fff0000</span> <span>==</span> <span>0x00070000</span> <span>{</span>\n                        <span>r0</span> <span>&amp;=</span> <span>0xffff</span>\n                <span>}</span>\n                <span>hr</span> <span>=</span> <span>syscall</span><span>.</span><span>Errno</span><span>(</span><span>r0</span><span>)</span>\n        <span>}</span>\n        <span>return</span>\n<span>}</span>\n</code></pre></div></div>\n\n<p>PowerShell provides <code>Get-HnsNamespace</code> to list available namespaces. These <em>are</em> the droids values I’ve been looking for to put in <code>config.json</code>! However, by default there are no cmdlets to create them. The installation PowerShell <a href=\"https://github.com/microsoft/Windows-Containers/blob/Main/helpful_tools/Install-ContainerdRuntime/install-containerd-runtime.ps1\">script</a> for <code>containerd</code> pulls in <a href=\"https://github.com/microsoft/SDN/blob/master/Kubernetes/windows/hns.psm1\">hns.psm1</a> for <code>containerd</code>, has a lot of interesting cmdlets, such as <code>New-HnsNetwork</code>, but not a cmdlet to create a namespace. There is also <a href=\"https://github.com/microsoft/SDN/blob/master/Kubernetes/windows/hns.v2.psm1\">hns.v2.psm1</a>, which does have <code>New-HnsNamespace</code>.</p>\n\n<div><div><pre><code><span>PS</span><span> </span><span>C:\\Users\\Administrator</span><span>&gt;</span><span> </span><span>curl.exe</span><span> </span><span>-o</span><span> </span><span>hns.v2.psm1</span><span> </span><span>-L</span><span> </span><span>https://raw.githubusercontent.com/microsoft/SDN/refs/heads/master/Kubernetes/windows/hns.v2.psm1</span><span>\n  </span><span>%</span><span> </span><span>Total</span><span>    </span><span>%</span><span> </span><span>Received</span><span> </span><span>%</span><span> </span><span>Xferd</span><span>  </span><span>Average</span><span> </span><span>Speed</span><span>   </span><span>Time</span><span>    </span><span>Time</span><span>     </span><span>Time</span><span>  </span><span>Current</span><span>\n                                 </span><span>Dload</span><span>  </span><span>Upload</span><span>   </span><span>Total</span><span>   </span><span>Spent</span><span>    </span><span>Left</span><span>  </span><span>Speed</span><span>\n</span><span>100</span><span> </span><span>89329</span><span>  </span><span>100</span><span> </span><span>89329</span><span>    </span><span>0</span><span>     </span><span>0</span><span>   </span><span>349</span><span>k</span><span>      </span><span>0</span><span> </span><span>--</span><span>:</span><span>--</span><span>:</span><span>--</span><span> </span><span>--</span><span>:</span><span>--</span><span>:</span><span>--</span><span> </span><span>--</span><span>:</span><span>--</span><span>:</span><span>--</span><span>  </span><span>353k</span><span>\n\n</span><span>PS</span><span> </span><span>C:\\Users\\Administrator</span><span>&gt;</span><span> </span><span>Import-Module</span><span> </span><span>.</span><span>\\hns.v2.psm1</span><span>\n</span><span>WARNING:</span><span> </span><span>The</span><span> </span><span>names</span><span> </span><span>of</span><span> </span><span>some</span><span> </span><span>imported</span><span> </span><span>commands</span><span> </span><span>from</span><span> </span><span>the</span><span> </span><span>module</span><span> </span><span>'hns.v2'</span><span> </span><span>include</span><span> </span><span>unapproved</span><span> </span><span>verbs</span><span> </span><span>that</span><span> </span><span>might</span><span> </span><span>make</span><span> </span><span>them</span><span> </span><span>less</span><span> </span><span>discoverable.</span><span> </span><span>To</span><span> </span><span>find</span><span> </span><span>the</span><span> </span><span>commands</span><span> </span><span>with</span><span> </span><span>unapproved</span><span> </span><span>verbs</span><span>,</span><span> </span><span>run</span><span> </span><span>the</span><span> </span><span>Import-Module</span><span> </span><span>command</span><span> </span><span>again</span><span> </span><span>with</span><span> </span><span>the</span><span> </span><span>Verbose</span><span> </span><span>parameter.</span><span> </span><span>For</span><span> </span><span>a</span><span> </span><span>list</span><span> </span><span>of</span><span> </span><span>approved</span><span> </span><span>verbs</span><span>,</span><span> </span><span>type</span><span> </span><span>Get-Verb.</span><span>\n\n</span><span>PS</span><span> </span><span>C:\\Users\\Administrator</span><span>&gt;</span><span> </span><span>New-HnsNamespace</span><span>\n</span><span>HcnCreateNamespace</span><span> </span><span>--</span><span> </span><span>HRESULT:</span><span> </span><span>2151350299.</span><span> </span><span>Result:</span><span> </span><span>{</span><span>\"Success\"</span><span>:</span><span>false</span><span>,</span><span>\"Error\"</span><span>:</span><span>\"Invalid JSON document string. &amp;#123;&amp;#123;CreateWithCompartment,UnknownField}}\"</span><span>,</span><span>\"ErrorCode\"</span><span>:</span><span>2151350299</span><span>}</span><span>\n</span><span>At</span><span> </span><span>C:\\Users\\Administrator\\hns.v2.psm1:2392</span><span> </span><span>char:13</span><span>\n</span><span>+</span><span>             </span><span>throw</span><span> </span><span>$errString</span><span>\n</span><span>+</span><span>             </span><span>~~~~~~~~~~~~~~~~</span><span>\n    </span><span>+</span><span> </span><span>CategoryInfo</span><span>          </span><span>:</span><span> </span><span>OperationStopped:</span><span> </span><span>(</span><span>HcnCreateNamesp...de</span><span>\":2151350299}:String) [], RuntimeException\n    + FullyQualifiedErrorId : HcnCreateNamespace -- HRESULT: 2151350299. Result: {\"</span><span>Success</span><span>\":false,\"</span><span>Error</span><span>\":\"</span><span>Invalid</span><span> </span><span>JSON</span><span> </span><span>document</span><span> </span><span>string.</span><span> </span><span>&amp;</span><span>#123;&amp;#123;CreateWithCompartment,UnknownField}}\",\"ErrorCode\":2151350299}</span><span>\n</span></code></pre></div></div>\n\n<p>With a lot of frustration, I decided to have a go at calling the Win32 API from OCaml. This resulted in <a href=\"https://github.com/mtelvers/hcn-namespace\">mtelvers/hcn-namespace</a>, which allows me to create the namespaces by running <code>hcn-namespace create</code>. These namespaces appear in the output from <code>Get-HnsNamespace</code> and work correctly in <code>config.json</code>.</p>\n\n<p>Run <code>hcn-namespace.exe create</code>, and then populate <code>\"networkNamespace\": \"&lt;GUID&gt;\"</code> with the GUID provided and run with <code>ctr run --rm -cni --config config.json</code>.</p>",
 9  "content_type": "html",
10  "author": {
11    "name": "Mark Elvers",
12    "email": "mark.elvers@tunbury.org",
13    "uri": null
14  },
15  "categories": [
16    "containerd",
17    "tunbury.org"
18  ],
19  "source": "https://www.tunbury.org/atom.xml"
20}