Can the raspberry boot to an LVM root partition?

Question

Before I start, I will say I have posted this on the Rpi forum as well. I don't normally cross-post but, having tried there and got nothing, I've come to my preferred knowledge centre.

I'm just looking into orgainsing a Pi with a big disk and plan to use LVM. I expect to find that the root partition needs to be outside any LVM scheme - I thought I'd just ask on here to confirm my thoughts.

The way I boot x86 using root-on-LVM is via an initrd that includes the necessary lvm2 hook - the way you make an initrd is to use the mkinitcpio tool.

What I've discovered via Google and experimentation: The linux package is what brings in mkinitcpio on x86. On the Rpi, the linux package is linux-raspberrypi and this does not have mkinitcpio as a dependency (it was actually explicitly removed https://github.com/archlinuxarm/PKGBUILDs/pull/218). I've also read plenty about people on various distros struggling to use an initrd on the Pi so perhaps it doesn't work anyway and that's the reason for mknintcpio not being used.

So, would anyone more knowledgeable care to share their experience with me? Is it possible to boot the Pi with a root partition that is on an LVM volume ?

(I am talking about the root partition here, I am not including the boot partition in this - that's on the sd card and I know that's where it has to be. I put everything else on another hard-drive connected to the usb.)

Answer 1

This explains how you can boot the Raspberry Pi into a system with the root partition on an LVM volume. LVM is Logical Volume Management.

Configuration

In order for early-boot to be able to access an LVM volume the kernel must be able to load the LVM module before the root partition is mounted. This requires that an initial ramdisk (initrd) is configured and added to the boot configuration. Fortunately, this is quite straightforward.

Kernel Support

To be able to use an initial ramdisk, the kernel must be built with initrd support included. The stock Raspberry Pi ArchLinux kernel is. However, to be sure, verify this:

# zgrep INITRD /proc/config.gz
CONFIG_BLK_DEV_INITRD=y

It is not necessary to build a custom kernel if the CONFIG_BLK_DEV_INITRD option is enabled. Building a custom kernel is beyond the scope of this answer.

In case there is no config.gz in /proc, first run this:

sudo modprobe configs

Build the initrd

Firstly, install the mkinitcpio package:

# pacman -S mkinitcpio

Next, edit /etc/mkinitcpio.conf to add the lvm2 hook; locate the HOOKS= line and amend accordingly:

HOOKS="base udev autodetect modconf block lvm2 filesystems keyboard fsck"

Then generate the initrd:

# mkinitcpio -g /boot/initrd

Amend the boot configuration

The Raspberry Pi's boot is configured in two files, /boot/config.txt and /boot/cmdline.txt. Both need to be modified to install the initrd.

First, add the following line to the end of the `/boot/config.txt' file:

initramfs initrd 0x01f00000

Note that, unlike the other items in this file, the initramfs entry does not contain an equals = sign.

Next, amend the /boot/cmdline.txt file to add

...root=/dev/mmcblk0p1 initrd=0x01f00000 rootfstype=ext4...

Also, append an 'rw' otherwise the rootfs gets mounted read-only.

Reboot

With the above configuration in place, re-boot. If all is well, the system should come back up as before, but will have used the initial ramdisk during boot. The next step is to move the current root partition onto an LVM volume.

Move Root partitition to LVM

For the purpose of this example, we assume that you have a spare disk connected to the RPi's USB that is recognised as /dev/sdaand has a partition on it for LVM, say /dev/sda1. We will copy the existing root partition into a new volume created inside LVM.

Set up LVM

Set up LVM: create a new volume group and, within it, a logical volume for the root partition:

# pvcreate /dev/sda1
# vgcreate storage /dev/sda1
# lvcreate -L 5G storage -n root

This creates a volume group called storage and adds the partition /dev/sda1 to it. It creates a 5Gb logical voulume called root which can then be formatted and used like a regular partiton:

# mkfs.ext4 /dev/storage/root
# mount /dev/storage/root /mnt

Copy the current root onto it:

# rsync --progress -axv / /mnt/

Configure the Boot process

Configuring the boot process is a simple matter of changing the location of the root partition in /boot/cmdline.txt. Change root=/dev/mmcblk0p1 so that it points to the new volume: root=/dev/storage/root.

Re-boot once more and, if all things go well, the system should reboot with its root partition on the LVM volume.

Answer 2

On raspbian it is possible to create the initramfs with the mkinitramfs command.

I have successfully boot on a lvm root partition with the following steps:

1. create the lvm partition

2. copy actual root partition on the lvm partition (I have restart with root partition ro and copy with the dd command)

   • create a lvm partition (bigger than actual root partition):
     sudo lvcreate -n root -L 5G 3TB
   • add ro option in /etc/fstab for root partition
     /dev/mapper/3TB-root / ext4 defaults,noatime,ro 0 1
   • add ro option in /boot/cmdline.txt
     dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait ro
   • reboot: root partition is now read only and can be copy safely:
     sudo dd bs=1M if=/dev/mmcblk0p2 of=/dev/3TB/root
   • increase size of partition:
     sudo e2fsck -f /dev/3TB/root && sudo resize2fs /dev/3TB/root
   • remount root file system in rw:
     sudo mount -o remount,rw /

3. chroot in the lvm root partition sudo mkdir /mnt/root sudo mount /dev/mapper/3TB-root /mnt/root sudo mount -t proc /proc /mnt/root/proc sudo mount -o bind /dev /mnt/root/dev sudo mount -o bind /dev/pts /mnt/root/dev/pts sudo mount -t sysfs /sys /mnt/root/sys sudo chroot /mnt/root

4. change /etc/fstab : replace /dev/mmcblk0p2 by the lvm path ( for me /dev/mapper/3TB-root) and remove ro option
5. add this patch https://bugs.debian.org/cgi-bin/bugreport.cgi?msg=10;filename=lvm2_wait-lvm.patch;att=1;bug=568838 to /usr/share/initramfs-tools/scripts/local-top/lvm2
6. add "copy_exec /bin/grep" in /usr/share/initramfs-tools/hooks/lvm2 (because grep is used in previous patch)
7. mount /boot
8. create the initramfs: sudo mkinitramfs -o /boot/initramfs.gz
9. look size of /boot/initramfs.gz and convert it in hexa
10. in /boot/config.txt add this: initramfs initramfs.gz 0xa00000
11. remove ro option in /boot/cmdline.txt and add this root=/dev/mapper/3TB-root initrd=0xa00000,0x43b692 where root replace /dev/mmcblk0p2 by the lvm path and 0x43b692 is the hexa size of /boot/initramfs.gz
12. reboot

Now you should be on the lvm root partition.

Answer 3

This answer is obsolete. Here is a simpler solution.

For development I like to take snapshots so I can easily revert to original installation without flashing the SD Card again. For this we have to setup LVM (Logical Volume Management) on the SD Card and move the root partition to it. With LVM it is also very easy to make backups on the running system. For all this you have to make the following only one time. So I've made this answer to summarize all the nice information on this page from today's view.

The problem is that we have to create an initramfs with LVM drivers so the raspi can boot and mount the root partition which is residing on the logical volume. Creating the initramfs must be done on an operating system with ARM processor of course. I can't do this with my PC, it's an intel one. So I must boot into the raspi with a temporary installation only for creating the initramfs. Afterwards we will remove this installation and expand the volume group with its place so we have the whole disk space for our working installation.

Prepare SD Card with LVM

Here I'll prepare a Raspian Lite image for example. Maybe you must have to install the lvm management tools lvm2.

Flash SD Card

pc ~$ sudo -Es
pc ~# dd if=Downloads/2017-11-29-raspbian-stretch-lite.img of=/dev/sdd bs=4M conv=fsync

Create additional LVM partition of 3 GB

pc ~# parted /dev/sdd print
[..]
Number  Start   End     Size    Type     File system  Flags
 1      4194kB  47,7MB  43,5MB  primary  fat32        lba
 2      48,2MB  1858MB  1810MB  primary  ext4

pc ~# parted /dev/sdd mkpart primary 1858MB 100%
pc ~# parted /dev/sdd set 3 lvm on
pc ~# pvcreate /dev/sdd3
pc ~# vgcreate rpi_vg /dev/sdd3
pc ~# lvcreate rpi_vg --name root_lv --size 3G
pc ~# mkfs.ext4 /dev/mapper/rpi_vg-root_lv
pc ~# e2label /dev/mapper/rpi_vg-root_lv rootfs

Mount the partitions and copy root partition to the LVM partition

pc ~# mkdir /mnt/{sdd1,sdd2,root_lv}
pc ~# mount /dev/sdd1 /mnt/sdd1
pc ~# mount /dev/sdd2 /mnt/sdd2
pc ~# mount /dev/mapper/rpi_vg-root_lv /mnt/root_lv
pc ~# rsync -ax /mnt/sdd2/ /mnt/root_lv/

Change device name in cmdline.txt for root partition to /dev/mmcblk0p2 and in etc/fstab to /dev/mmcblk0p1 and /dev/mmcblk0p2 because of this Problem with device names for root partition.

Clean up

pc ~# umount /mnt/{sdd1,sdd2,root_lv}
pc ~# e2fsck -f /dev/sdd2
pc ~# e2fsck -f /dev/mapper/rpi_vg-root_lv
pc ~# rmdir /mnt/{sdd1,sdd2,root_lv}
pc ~# exit
pc ~$

Create initramfs with temporary installation

Boot the SD Card in a raspi.

After login to the raspberrypi first install lvm2 and reboot to take affect. Be patient if it takes sometimes a little bit longer. There are many mounts to unmount with timeout.

raspberrypi ~$ sudo apt install lvm2
raspberrypi ~$ sudo systemctl reboot

After login chroot into root_lv and create the initramfs with mkinitramfs.

pi@raspberrypi ~$ ls /dev/mapper/rpi_vg-root_lv
/dev/mapper/rpi_vg-root_lv

Give the kernels pseudo files to the chrooted system.

raspberrypi ~$ sudo -Es
raspberrypi ~# mount /dev/mapper/rpi_vg-root_lv /mnt/
raspberrypi ~# cd /mnt
raspberrypi ~# mount /dev/mmcblk0p1 boot/
raspberrypi ~# mount -t proc /proc proc
raspberrypi ~# mount -o bind /dev dev
raspberrypi ~# mount -o bind /dev/pts dev/pts
raspberrypi ~# mount -t sysfs /sys sys
raspberrypi ~# chroot /mnt/

Install `lvm2´ also in this system:

raspberrypi ~# apt install lvm2

Add this line to /boot/config.txt without =!:

initramfs initramfs.gz

Then correct device names for root partition like this:

raspberrypi ~# cat /boot/cmdline.txt
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/mapper/rpi_vg-root_lv rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait

raspberrypi ~# cat /etc/fstab
proc                        /proc           proc    defaults          0       0
/dev/mmcblk0p1              /boot           vfat    defaults          0       2
/dev/mapper/rpi_vg-root_lv  /               ext4    defaults,noatime  0       1
raspberrypi ~#

Now we are ready to create the initramfs:

raspberrypi ~# mkinitramfs -o /boot/initramfs.gz

Reboot

raspberrypi ~# exit
raspberrypi ~# exit
raspberrypi ~$ sudo systemctl reboot

Work with the final installation

After login check if we are mounted on /dev/mapper/rpi_vg-root_lv:

raspberrypi ~$ lsblk
NAME               MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
mmcblk0            179:0    0  7.5G  0 disk
├─mmcblk0p1        179:1    0 41.5M  0 part /boot
├─mmcblk0p2        179:2    0  1.7G  0 part
└─mmcblk0p3        179:3    0  5.8G  0 part
  └─rpi_vg-root_lv 254:0    0    3G  0 lvm  /
raspberrypi ~$

Now we can add the partition from the temporary installation to the volume group, so we can use it to expand our logical volume for this system or for snapshots.

raspberrypi ~$ sudo -Es
raspberrypi ~# pvcreate /dev/mmcblk0p2
raspberrypi ~# vgextend rpi_vg /dev/mmcblk0p2

raspberrypi ~# pvs
  PV             VG     Fmt  Attr PSize PFree
  /dev/mmcblk0p2 rpi_vg lvm2 a--  1.68g 1.68g
  /dev/mmcblk0p3 rpi_vg lvm2 a--  5.78g 2.78g
raspberrypi ~#

But if you have enough space on your SD Card it could be a good idea to leave the installation on /dev/mmcblk0p2 untouched. Sometimes after an update on the working installation on /dev/mmcblk0p3 I forgot to sudo mkinitramfs -o /boot/initramfs.gz to also update the initramfs, so boot fails. Rebooting into rescue partition on /dev/mmcblk0p2 and recreating initramfs helps.

Update the installation.

raspberrypi ~# apt update
raspberrypi ~# apt full-upgrade
raspberrypi ~# mkinitramfs -o /boot/initramfs.gz
raspberrypi ~# apt clean
raspberrypi ~# exit
raspberrypi ~$ sudo systemctl reboot

Take snapshots and backups

Login. Now we should take our first snapshot from this base installation so we can always revert to it:

raspberrypi ~$ sudo lvcreate --snapshot --name rpi_base --size 3G rpi_vg/root_lv
raspberrypi ~$ sudo lvs
  LV       VG     Attr       LSize Pool Origin  Data%  Meta%  Move Log Cpy%Sync Convert
  root_lv  rpi_vg owi-aos--- 3.00g
  rpi_base rpi_vg swi-a-s--- 3.00g      root_lv 0.00
raspberrypi ~$

Revert and remove(!) snapshot. You should create it afterwards immediately again.

raspberrypi ~$ sudo lvconvert --merge rpi_vg/rpi_base
raspberrypi ~$ sudo systemctl reboot

Be patient! It takes a long time. Never switch off your raspi.

Login

raspberrypi ~$ sudo lvcreate --snapshot --name rpi_base --size 3G rpi_vg/root_lv

To commit simply delete the snapshot:

raspberrypi ~$ sudo lvremove rpi_vg/rpi_base

To make a backup:
simply make a snapshot, mount it and backup it, e.g. with the base installation

raspberrypi ~$ sudo mount /dev/mapper/rpi_vg-rpi_base /mnt/
raspberrypi ~$ tar -czf - -C /mnt/ ./ | ssh backup-server dd of=raspi.tar.gz bs=10k 

Answer 4

There are two ways you can use initramfs with kernel. It can integrated into kernel image or loaded as a separate file.

Embedded initramfs

First mode is definitely working since it is being used by kernel_emergency.img. In order to use this method you have to compile your own kernel image using following config options:

CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE="/path/to/your/initramfs/content"

You can find some more information on how to do this in this answer.

External initramfs

The second mode is to use separate initramfs file. This has an advantage of not requiring recompiling the kernel each time you want to change your initramfs. You need some help from bootloader in order to use it, though.

Back in July 2012 I was able to cooperate with Dom from Broadcom in fixing some problems with RaspberryPi bootloader and this made it possible to use external initramfs without problems. Unless there is some regression, it should still be working.

Unfortunately, you also need custom kernel in order to use this mode. CONFIG_BLK_DEV_INITRD=y should be enabled. You should put your iniramfs file on /boot/ partition and add this line to config.txt:

initramfs initramfs.gz

(initramfs.gz should be changed to your filename). Note there is no = sign in this option.

Of course you also have to create actual initramfs file. If your initramfs content is in directory /tmp/initramfs, you should run following commands:

cd /tmp/initramfs
find . -print0 | cpio --null -ov --format=newc | gzip -9 > /boot/initramfs.gz

The hard part

Regardless of the method used, you have to create your initramfs content. While all I wrote above is distribution agnostic, creating actual initramfs is somehow dependent on the distribution you are using. I don't know Arch Linux and I didn't use any automated method. I do this manually:

1. Create dev directory and populate it with essential files (console, null, tty, sda*). You can copy them from your /dev/ directory.
2. Create some essential directories (/proc/, /sys/, /mnt/root).
3. Compile busybox with lvm support and install it inside of your initramfs directory.

4. Write start script (named init) with something like:

   #!/bin/busybox sh
   
   mount -t proc none /proc
   mount -t sysfs none /sys
   
   lvm vgscan --mknodes
   lvm vgchange -ay VGNAME 
   mount -o ro /dev/VGNAME/LVNAME /mnt/root
   
   umount /proc
   umount /sys
   exec switch_root /mnt/root /sbin/init
   

Note: that this instruction is put here from the back of my head so it may not be complete. It's here mainly so that you can understand the idea. It is also possible that you have to do some more things in your distribution.