Thicket data repository for the EEG
anil.recoil.org
1{
2 "id": "https://www.tunbury.org/2025/05/01/removing-mdadm",
3 "title": "Reconfiguring a system with an mdadm RAID5 root",
4 "link": "https://www.tunbury.org/2025/05/01/removing-mdadm/",
5 "updated": "2025-05-01T12:00:00",
6 "published": "2025-05-01T12:00:00",
7 "summary": "Cloud providers automatically configure their machines as they expect you to use them. For example, a machine with 4 x 8T disks might come configured with an mdadm RAID5 array spanning the disks. This may be what most people want, but we don’t want this configuration, as we want to see the bare disks. Given you have only a serial console (over SSH) and no access to the cloud-init environment, how do you boot the machine in a different configuration?",
8 "content": "<p>Cloud providers automatically configure their machines as they expect you to use them. For example, a machine with 4 x 8T disks might come configured with an mdadm RAID5 array spanning the disks. This may be what most people want, but we don’t want this configuration, as we want to see the bare disks. Given you have only a serial console (over SSH) and no access to the cloud-init environment, how do you boot the machine in a different configuration?</p>\n\n<p>Example configuration:</p>\n\n<div><div><pre><code>$ lsblk\nNAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS\nfd0 2:0 1 4K 0 disk\nsda 8:0 0 4G 0 disk\n├─sda1 8:1 0 512M 0 part /boot/efi\n└─sda2 8:2 0 3.5G 0 part\n └─md0 9:0 0 10.5G 0 raid5 /\nsdb 8:16 0 4G 0 disk\n└─sdb1 8:17 0 4G 0 part\n └─md0 9:0 0 10.5G 0 raid5 /\nsdc 8:32 0 4G 0 disk\n└─sdc1 8:33 0 4G 0 part\n └─md0 9:0 0 10.5G 0 raid5 /\nsdd 8:48 0 4G 0 disk\n└─sdd1 8:49 0 4G 0 part\n └─md0 9:0 0 10.5G 0 raid5 /\n</code></pre></div></div>\n\n<p>My initial approach was to create a tmpfs root filesystem and then use <code>pivot_root</code> to switch it. This worked except <code>/dev/md0</code> was still busy, so I could not unmount it.</p>\n\n<p>It occurred to me that I could remove one of the partitions from the RAID5 set and use that as the new root disk. <code>mdadm --fail /dev/md0 /dev/sda2</code>, followed by <code>mdadm --remove /dev/md0 /dev/sda2</code> frees up a disk. <code>debootstrap</code> can then be used to install Ubuntu on the partition. As we have a working system, we can preserve the key configuration settings such as <code>/etc/hostname</code>, <code>/etc/netplan</code>, <code>/etc/fstab</code> etc by just copying them from <code>/etc</code> to <code>/mnt/etc</code>. Unfortunately, Ansible’s copy module does not preserve ownership. Therefore, I used <code>rsync</code> instead. <code>/etc/fstab</code> must be edited to reflect the new root partition.</p>\n\n<p>Lastly, run <code>grub-install</code> using <code>chroot</code> to the new environment and reboot.</p>\n\n<div><div><pre><code># lsblk\nNAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS\nfd0 2:0 1 4K 0 disk\nsda 8:0 0 4G 0 disk\n├─sda1 8:1 0 512M 0 part /boot/efi\n└─sda2 8:2 0 3.5G 0 part /\nsdb 8:16 0 4G 0 disk\n└─sdb1 8:17 0 4G 0 part\nsdc 8:32 0 4G 0 disk\n└─sdc1 8:33 0 4G 0 part\nsdd 8:48 0 4G 0 disk\n└─sdd1 8:49 0 4G 0 part\n</code></pre></div></div>\n\n<p>The redundant RAID5 partitions can be removed with <code>wipefs -af /dev/sd[b-d]</code></p>\n\n<p>I have wrapped all the steps in an Ansible <a href=\"https://gist.github.com/mtelvers/1fe3571830d982eb8adbcf5a513edb2c\">playbook</a>, which is available as a GitHub gist.</p>\n\n<h1>Addendum</h1>\n\n<p>I had tested this in QEMU with EFI under the assumption that a newly provisioned cloud machine would use EFI. However, when I ran the script against the machine, I found it used a legacy bootloader, and it was even more complicated than I had envisioned, as there were three separate MDADM arrays in place:</p>\n\n<div><div><pre><code># cat /proc/mdstat \nPersonalities : [raid1] [raid6] [raid5] [raid4] [raid0] [raid10] \nmd2 : active raid5 sdb4[0] sdd4[2] sda4[4] sdc4[1]\n 34252403712 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]\n bitmap: 2/86 pages [8KB], 65536KB chunk\n\nmd1 : active raid5 sdd3[1] sda3[2] sdc3[0] sdb3[4]\n 61381632 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]\n \nmd0 : active raid1 sdd2[1] sda2[2] sdb2[3] sdc2[0]\n 523264 blocks super 1.2 [4/4] [UUUU]\n \nunused devices: <none>\n</code></pre></div></div>\n\n<p>With <code>lsblk</code> showing four disks each configured as below:</p>\n\n<div><div><pre><code>NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS\nsda 8:0 0 10.9T 0 disk \n├─sda1 8:1 0 1M 0 part \n├─sda2 8:2 0 512M 0 part \n│ └─md0 9:0 0 511M 0 raid1 \n│ └─md0p1 259:0 0 506M 0 part /boot\n├─sda3 8:3 0 19.5G 0 part \n│ └─md1 9:1 0 58.5G 0 raid5 \n│ └─md1p1 259:1 0 58.5G 0 part /\n├─sda4 8:4 0 10.6T 0 part \n│ └─md2 9:2 0 31.9T 0 raid5 \n│ └─md2p1 259:2 0 31.9T 0 part /data\n└─sda5 8:5 0 512M 0 part [SWAP]\n</code></pre></div></div>\n\n<p>The boot device is a RAID1 mirror (four copies), so removing one of these copies is no issue. There is also a 1MB BIOS boot partition first to give some space for GRUB. The root device was RAID5 as I had anticipated.</p>\n\n<p>The playbook could be adapted: double up on the <code>mdadm</code> commands to break two arrays, update two entries in <code>/etc/fstab</code> and use <code>grub-pc</code> rather than <code>grub-efi-amd64</code>. The updated playbook is <a href=\"https://gist.github.com/mtelvers/ba3b7a5974b50422e2c2e594bed0bdb2\">here</a>.</p>\n\n<p>For testing, I installed Ubuntu using this <a href=\"https://gist.github.com/mtelvers/d2d333bf5c9bd94cb905488667f0cae1\">script</a> to simulate the VM.</p>\n\n<p>Improvements could be made, as <code>/boot</code> could be merged into <code>/</code> as there is no reason to separate them when not using EFI. There never <em>needed</em> to be a <code>/boot</code> as GRUB2 will boot a RAID5 MDADM.</p>\n\n<p>The system is a pretty minimal installation of Ubuntu, a more typical set of tools could be installed with:</p>\n\n<div><div><pre><code>apt install ubuntu-standard\n</code></pre></div></div>",
9 "content_type": "html",
10 "author": {
11 "name": "Mark Elvers",
12 "email": "mark.elvers@tunbury.org",
13 "uri": null
14 },
15 "categories": [
16 "mdadm,ubuntu",
17 "tunbury.org"
18 ],
19 "source": "https://www.tunbury.org/atom.xml"
20}