Thicket data repository for the EEG
anil.recoil.org
1{
2 "id": "https://www.tunbury.org/2025/07/07/refs-monteverde",
3 "title": "ReFS, EEG Intern and Monteverde",
4 "link": "https://www.tunbury.org/2025/07/07/refs-monteverde/",
5 "updated": "2025-07-07T00:00:00",
6 "published": "2025-07-07T00:00:00",
7 "summary": "In addition to the post from last week covering BON in a Box and OCaml Functors, below are some additional notes.",
8 "content": "<p>In addition to the post from last week covering <a href=\"https://www.tunbury.org/2025/07/02/bon-in-a-box/\">BON in a Box</a> and <a href=\"https://www.tunbury.org/2025/07/01/ocaml-functors/\">OCaml Functors</a>, below are some additional notes.</p>\n\n<h1>Resilient File System, ReFS</h1>\n\n<p>I have previously stated that <a href=\"https://www.tunbury.org/windows-reflinks\">ReFS</a> supports 1 million hard links per file; however, this is not the case. The maximum is considerably lower at 8191. That’s eight times more than NTFS, but still not very many.</p>\n\n<div><div><pre><code><span>PS</span><span> </span><span>D:\\</span><span>></span><span> </span><span>touch</span><span> </span><span>foo</span><span>\n</span><span>PS</span><span> </span><span>D:\\</span><span>></span><span> </span><span>foreach</span><span> </span><span>(</span><span>$i</span><span> </span><span>in</span><span> </span><span>1</span><span>..</span><span>8192</span><span>)</span><span> </span><span>{</span><span>\n</span><span>>></span><span> </span><span>New-Item</span><span> </span><span>-ItemType</span><span> </span><span>HardLink</span><span> </span><span>-Path</span><span> </span><span>\"foo-</span><span>$i</span><span>\"</span><span> </span><span>-Target</span><span> </span><span>\"foo\"</span><span>\n</span><span>>></span><span> </span><span>}</span><span>\n\n\n </span><span>Directory:</span><span> </span><span>D:\\</span><span>\n\n\n</span><span>Mode</span><span> </span><span>LastWriteTime</span><span> </span><span>Length</span><span> </span><span>Name</span><span>\n</span><span>----</span><span> </span><span>-------------</span><span> </span><span>------</span><span> </span><span>----</span><span>\n</span><span>-a</span><span>----</span><span> </span><span>07</span><span>/07/2025</span><span> </span><span>01:00</span><span> </span><span>0</span><span> </span><span>foo-1</span><span>\n</span><span>-a</span><span>----</span><span> </span><span>07</span><span>/07/2025</span><span> </span><span>01:00</span><span> </span><span>0</span><span> </span><span>foo-2</span><span>\n</span><span>-a</span><span>----</span><span> </span><span>07</span><span>/07/2025</span><span> </span><span>01:00</span><span> </span><span>0</span><span> </span><span>foo-3</span><span>\n</span><span>-a</span><span>----</span><span> </span><span>07</span><span>/07/2025</span><span> </span><span>01:00</span><span> </span><span>0</span><span> </span><span>foo-4</span><span>\n</span><span>...</span><span>\n</span><span>-a</span><span>----</span><span> </span><span>07</span><span>/07/2025</span><span> </span><span>01:00</span><span> </span><span>0</span><span> </span><span>foo-8190</span><span>\n</span><span>-a</span><span>----</span><span> </span><span>07</span><span>/07/2025</span><span> </span><span>01:00</span><span> </span><span>0</span><span> </span><span>foo-8191</span><span>\n</span><span>New-Item</span><span> </span><span>:</span><span> </span><span>An</span><span> </span><span>attempt</span><span> </span><span>was</span><span> </span><span>made</span><span> </span><span>to</span><span> </span><span>create</span><span> </span><span>more</span><span> </span><span>links</span><span> </span><span>on</span><span> </span><span>a</span><span> </span><span>file</span><span> </span><span>than</span><span> </span><span>the</span><span> </span><span>file</span><span> </span><span>system</span><span> </span><span>supports</span><span>\n</span><span>At</span><span> </span><span>line:2</span><span> </span><span>char:5</span><span>\n</span><span>+</span><span> </span><span>New-Item</span><span> </span><span>-ItemType</span><span> </span><span>HardLink</span><span> </span><span>-Path</span><span> </span><span>\"foo-</span><span>$i</span><span>\"</span><span> </span><span>-Target</span><span> </span><span>\"foo\"</span><span>\n</span><span>+</span><span> </span><span>~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~</span><span>\n </span><span>+</span><span> </span><span>CategoryInfo</span><span> </span><span>:</span><span> </span><span>NotSpecified:</span><span> </span><span>(:)</span><span> </span><span>[</span><span>New</span><span>-Item</span><span>],</span><span> </span><span>Win32Exception</span><span>\n </span><span>+</span><span> </span><span>FullyQualifiedErrorId</span><span> </span><span>:</span><span> </span><span>System.ComponentModel.Win32Exception</span><span>,</span><span>Microsoft.PowerShell.Commands.NewItemCommand</span><span>\n</span></code></pre></div></div>\n\n<p>I had also investigated ReFS block cloning, which removed the requirement to create hard links, and wrote a <a href=\"https://github.com/mtelvers/ReFS-Clone\">ReFS-clone</a> tool for Windows Server 2022. This works well until containerd is used to bind mount a directory on the volume. Once this has happened, attempts to create a block clone fail. To exclude my code as the root cause, I have tried Windows Server 2025, where commands such as <code>copy</code> and <code>robocopy</code> automatically perform block clones. Block cloning can be restored by rebooting the machine. I note that restarting containerd is not sufficient.</p>\n\n<p>Removing files and folders on ReFS is impressively fast; however, this comes at a cost: freeing the blocks is a background activity that may take some time to be scheduled.</p>\n\n<h1>File system performance with a focus on ZFS</h1>\n\n<p>Several EEG interns started last week with this <a href=\"https://anil.recoil.org/ideas/zfs-filesystem-perf\">project</a> under my supervision. In brief, we will examine file system performance on the filesystems supported by <a href=\"https://github.com/ocurrent/obuilder\">OBuilder</a> before conducting more detailed investigations into factors affecting ZFS performance.</p>\n\n<h1>Monteverde</h1>\n\n<p>monteverde.cl.cam.ac.uk, has been installed in the rack. It has two AMD EPYC 9965 192-Core Processors, giving a total of 384 cores and 768 threads and 3TB of RAM.</p>\n\n<p><img alt=\"\" src=\"https://www.tunbury.org/images/monteverde.jpg\"></p>\n\n<p>From the logs, there are still some teething issues:</p>\n\n<div><div><pre><code>[130451.620482] Large kmem_alloc(98304, 0x1000), please file an issue at:\n https://github.com/openzfs/zfs/issues/new\n[130451.620486] CPU: 51 UID: 0 PID: 8594 Comm: txg_sync Tainted: P O 6.14.0-23-generic #23-Ubuntu\n[130451.620488] Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE\n[130451.620489] Hardware name: Dell Inc. PowerEdge R7725/0KRFPX, BIOS 1.1.3 02/25/2025\n[130451.620490] Call Trace:\n[130451.620490] <TASK>\n[130451.620492] show_stack+0x49/0x60\n[130451.620493] dump_stack_lvl+0x5f/0x90\n[130451.620495] dump_stack+0x10/0x18\n[130451.620497] spl_kmem_alloc_impl.cold+0x17/0x1c [spl]\n[130451.620503] spl_kmem_zalloc+0x19/0x30 [spl]\n[130451.620508] multilist_create_impl+0x3f/0xc0 [zfs]\n[130451.620586] multilist_create+0x31/0x50 [zfs]\n[130451.620650] dmu_objset_sync+0x4c4/0x4d0 [zfs]\n[130451.620741] dsl_pool_sync_mos+0x34/0xc0 [zfs]\n[130451.620832] dsl_pool_sync+0x3c1/0x420 [zfs]\n[130451.620910] spa_sync_iterate_to_convergence+0xda/0x220 [zfs]\n[130451.620990] spa_sync+0x333/0x660 [zfs]\n[130451.621056] txg_sync_thread+0x1f5/0x270 [zfs]\n[130451.621137] ? __pfx_txg_sync_thread+0x10/0x10 [zfs]\n[130451.621207] ? __pfx_thread_generic_wrapper+0x10/0x10 [spl]\n[130451.621213] thread_generic_wrapper+0x5b/0x70 [spl]\n[130451.621217] kthread+0xf9/0x230\n[130451.621219] ? __pfx_kthread+0x10/0x10\n[130451.621221] ret_from_fork+0x44/0x70\n[130451.621223] ? __pfx_kthread+0x10/0x10\n[130451.621224] ret_from_fork_asm+0x1a/0x30\n[130451.621226] </TASK>\n</code></pre></div></div>",
9 "content_type": "html",
10 "author": {
11 "name": "Mark Elvers",
12 "email": "mark.elvers@tunbury.org",
13 "uri": null
14 },
15 "categories": [
16 "refs",
17 "tunbury.org"
18 ],
19 "source": "https://www.tunbury.org/atom.xml"
20}