Thicket data repository for the EEG
anil.recoil.org
1{
2 "id": "https://www.tunbury.org/2025/07/15/reflink-copy",
3 "title": "Reflink Copy",
4 "link": "https://www.tunbury.org/2025/07/15/reflink-copy/",
5 "updated": "2025-07-15T00:00:00",
6 "published": "2025-07-15T00:00:00",
7 "summary": "I hadn’t intended to write another post about traversing a directory structure or even thinking about it again, but weirdly, it just kept coming up again!",
8 "content": "<p>I hadn’t intended to write another <a href=\"https://www.tunbury.org/2025/07/08/unix-or-sys/\">post</a> about traversing a directory structure or even thinking about it again, but weirdly, it just kept coming up again!</p>\n\n<p>Firstly, Patrick mentioned <code>Eio.Path.read_dir</code> and Anil mentioned <a href=\"https://tavianator.com/2023/bfs_3.0.html\">bfs</a>. Then Becky commented about XFS reflink performance, and I commented that the single-threaded nature of <code>cp -r --reflink=always</code> was probably hurting our <a href=\"https://github.com/ocurrent/obuilder\">obuilder</a> performance tests.</p>\n\n<p>Obuilder is written in LWT, which has <code>Lwt_unix.readdir</code>. What if we had a pool of threads that would traverse the directory structure in parallel and create a reflinked copy?</p>\n\n<p>Creating a reflink couldn’t be easier. There’s an <code>ioctl</code> call that <em>just</em> does it. Such a contrast to the ReFS copy-on-write implementation on Windows!</p>\n\n<div><div><pre><code><span>#include</span> <span><caml/mlvalues.h></span><span>\n#include</span> <span><caml/memory.h></span><span>\n#include</span> <span><caml/unixsupport.h></span><span>\n#include</span> <span><sys/ioctl.h></span><span>\n#include</span> <span><errno.h></span><span>\n</span>\n<span>#ifndef FICLONE\n#define FICLONE 0x40049409\n#endif\n</span>\n<span>value</span> <span>caml_ioctl_ficlone</span><span>(</span><span>value</span> <span>dst_fd</span><span>,</span> <span>value</span> <span>src_fd</span><span>)</span> <span>{</span>\n <span>CAMLparam2</span><span>(</span><span>dst_fd</span><span>,</span> <span>src_fd</span><span>);</span>\n <span>int</span> <span>result</span><span>;</span>\n\n <span>result</span> <span>=</span> <span>ioctl</span><span>(</span><span>Int_val</span><span>(</span><span>dst_fd</span><span>),</span> <span>FICLONE</span><span>,</span> <span>Int_val</span><span>(</span><span>src_fd</span><span>));</span>\n\n <span>if</span> <span>(</span><span>result</span> <span>==</span> <span>-</span><span>1</span><span>)</span> <span>{</span>\n <span>uerror</span><span>(</span><span>\"ioctl_ficlone\"</span><span>,</span> <span>Nothing</span><span>);</span>\n <span>}</span>\n\n <span>CAMLreturn</span><span>(</span><span>Val_int</span><span>(</span><span>result</span><span>));</span>\n<span>}</span>\n</code></pre></div></div>\n\n<p>We can write a reflink copy function as shown below. (Excuse my error handling.) Interestingly, points to note: the permissions set via <code>Unix.openfile</code> are filtered through umask, and you need to <code>Unix.fchown</code> before <code>Unix.fchmod</code> if you want to set the suid bit set.</p>\n\n<div><div><pre><code><span>external</span> <span>ioctl_ficlone</span> <span>:</span> <span>Unix</span><span>.</span><span>file_descr</span> <span>-></span> <span>Unix</span><span>.</span><span>file_descr</span> <span>-></span> <span>int</span> <span>=</span> <span>\"caml_ioctl_ficlone\"</span>\n\n<span>let</span> <span>copy_file</span> <span>src</span> <span>dst</span> <span>stat</span> <span>=</span>\n <span>let</span> <span>src_fd</span> <span>=</span> <span>Unix</span><span>.</span><span>openfile</span> <span>src</span> <span>[</span><span>O_RDONLY</span><span>]</span> <span>0</span> <span>in</span>\n <span>let</span> <span>dst_fd</span> <span>=</span> <span>Unix</span><span>.</span><span>openfile</span> <span>dst</span> <span>[</span><span>O_WRONLY</span><span>;</span> <span>O_CREAT</span><span>;</span> <span>O_TRUNC</span><span>]</span> <span>0o600</span> <span>in</span>\n <span>let</span> <span>_</span> <span>=</span> <span>ioctl_ficlone</span> <span>dst_fd</span> <span>src_fd</span> <span>in</span>\n <span>Unix</span><span>.</span><span>fchown</span> <span>dst_fd</span> <span>stat</span><span>.</span><span>st_uid</span> <span>stat</span><span>.</span><span>st_gid</span><span>;</span>\n <span>Unix</span><span>.</span><span>fchmod</span> <span>dst_fd</span> <span>stat</span><span>.</span><span>st_perm</span><span>;</span>\n <span>Unix</span><span>.</span><span>close</span> <span>src_fd</span><span>;</span>\n <span>Unix</span><span>.</span><span>close</span> <span>dst_fd</span><span>;</span>\n</code></pre></div></div>\n\n<p>My LWT code created a list of all the files in a directory and then processed the list with <code>Lwt_list.map_s</code> (serially), returning promises for all the file operations and creating threads for new directory operations up to a defined maximum (8). If there was no thread capacity, it just recursed in the current thread. Copying a root filesystem, this gave me threads for <code>var</code>, <code>usr</code>, etc, just as we’d want. Wow! This was slow. Nearly 4 minutes to reflink 1.7GB!</p>\n\n<p>What about using the threads library rather than LWT threads? This appears significantly better, bringing the execution time down to 40 seconds. However, I think a lot of that was down to my (bad) LWT implementation vs my somewhat better threads implementation.</p>\n\n<p>At this point, I should probably note that <code>cp -r --reflink always</code> on 1.7GB, 116,000 files takes 8.5 seconds on my machine using a loopback XFS. A sequential OCaml version, without the overhead of threads or any need to maintain a list of work to do, takes 9.0 seconds.</p>\n\n<p>Giving up and getting on with other things was very tempting, but there was that nagging feeling of not bottoming out the problem.</p>\n\n<p>Using OCaml Multicore, we can write a true multi-threaded version. I took a slightly different approach, having a work queue of directories to process, and N worker threads taking work from the queue.</p>\n\n<div><div><pre><code>Main Process: Starts with root directory\n ↓\nWorkQueue: [process_dir(/root)]\n ↓\nDomain 1: Takes work → processes files → adds subdirs to queue\nDomain 2: Takes work → processes files → adds subdirs to queue\nDomain 3: Takes work → processes files → adds subdirs to queue\n ↓\nWorkQueue: [process_dir(/root/usr), process_dir(/root/var), ...]\n</code></pre></div></div>\n\n<p>Below is a table showing the performance when using multiple threads compared to the baseline operation of <code>cp</code> and a sequential copy in OCaml.</p>\n\n\n\n \n \n Copy command\n Duration (sec)\n \n \n \n \n cp -r –reflink=always\n 8.49\n \n \n Sequential\n 8.80\n \n \n 2 domains\n 5.45\n \n \n 4 domains\n 3.28\n \n \n 6 domains\n 3.43\n \n \n 8 domains\n 5.24\n \n \n 10 domains\n 9.07\n \n \n\n\n<p>The code is available on GitHub in <a href=\"https://github.com/mtelvers/reflink\">mtelvers/reflink</a>.</p>",
9 "content_type": "html",
10 "author": {
11 "name": "Mark Elvers",
12 "email": "mark.elvers@tunbury.org",
13 "uri": null
14 },
15 "categories": [
16 "ocaml"
17 ],
18 "source": "https://www.tunbury.org/atom.xml"
19}