Illegitimi non carborundum


The missing link

So I've just pushed a large update to Mageia Cauldron that implements everything needed to merge the / and /usr file systems for certain key folders.This was a change that was already widely supported by those in the know, but after other distros (e.g. Fedora) doing similar moves I proposed this as a Feature for Mageia 3.

To give some background, most binaries and libraries on a typical linux system are installed in the /usr folder tree. This is typically made up from the /usr/bin, /usr/sbin and the /usr/lib[64] folders for binaries, system binaries (the "s" used to stand for static in the dim and distant past but I think "system" was retrofitted later - to be honest a bin and sbin split is kinda silly too but that's another story) and libraries respectively.

Sadly, it's also quite desirable to have /usr on a separate partition in many cases. This can be for security (i.e. you can mount /usr readonly and do a hash check to detect any changes etc) or for reuse (i.e. you can share the /usr partition over the network to create a network booting cluster of machines. But because /usr is a separate partition, this means that the userspace needs to be able to mount the partition. This lead to certain binaries (like "mount") and certainly kernel modules (like "btrfs", "xfs" etc.) all being moved to the / filesystem (i.e. outside of /usr) such that they can be used to actually mount /usr. Of course things got even more complicated when you introduced things like encrypted partitions and LVM. All the binaries and libs needed to work with such /usr partitions also had to be moved to /. This meant that the split of binaries was very ad-hoc. As some new tool or partition type was supported, more and more stuff had to be moved. This moving had two main disadvantages - it made sharing /usr on the network less useful as several key binaries and libs were simply not there. Also even with a "secure" /usr, mounted read-only with hash checks etc, significant risk of poisoning would still exist for the binaries kept in /

So, a more modern approach these days is to use an initrd to mount /usr for you rather than rely on the installed system. A modern initrd generator, like dracut can detect your /usr mount and ensure the necessary binaries and libs are copied to an initrd tailored to the system in question. This allows different distros to share the same solutions and work better collaboratively rather than just "solve" (paper over) the same problems over and over again by moving binaries and libs around as needed.

So the main aim of this process was to remove the /bin /sbin /lib and /lib64 folders, moving all their contents to their respective folders in /usr and then just symlink the folders to allow anything using an absolute path to carry on working in blissful ignorance. This vastly simplifies things from a packaging perspective and allows much less in the way of hacks and quirks while making /usr genuinely useful and more secure again! Yay! The only disadvantage is that you now must use an initrd to boot a system with a separate /usr. This is generally not a huge deal as the reasons for doing a separate /usr mean that using an initrd is not a problem. For Mageia specifically we've never really supported booting without an initrd anyway so this is not any drawback for us.


The actual process of rolling out this change was actually quite simple in the end. Fedora already wrote a patch to RPM that allowed for a special filesystem check to be added to the filesystem RPM that prevented it from installing on systems which an incorrect folder layout. Then all that remained was to do a few urpmq/urpmf queries to find out all the packages where a file in / had the same name as a file in /usr. In most cases the affected packages would "self conflict" i.e. they'd provide a /bin/foo and a /usr/bin/foo (the latter typically being a convenience symlink to the former). Ultimately we had about forty packages that needed updating.

In order to not break the build system, I updated all of these packages manually first, then build them all in a clean chroot for both i586 and x86_64 architecture. Each of these packages depended on a new filesystem package. This filesystem package made sure to implement the filesystem check added via the rpm patch. By doing this, we ensured that no user could install the packages unless they have performed a "conversion" via their initrd before / is mounted as the rootfs. By building manually it also meant that the chroot used for automatically building RPMs did not get broken due to it not being able to install a new filesystem rpm.

The conversion itself was handled by an existing module in dracut. I took this module and augmented it with a handful of small quirks to deal with some packaging oddness in Mageia (thinking ahead to how we'll cope with a Mageia 2 -> Mageia 3 upgrade procedure).

After this was all done, I tested the whole process in various VMs and ironed out the (minor) issues before pushing out the new packages to the world. The only thing that went wrong was that I forgot the chroots were created by the rpm version provided by Mageia 1. This meant that the new check in the filesystem rpm was not recognised (as the Mageia 1 RPM didn't have the patch) and thus the chroots couldn't be built. This was fairly painless to fix and we were soon cooking on gas.

Several others have now gone through this process and all seems to be going very smoothly so far!

So that's two of my proposed features for Mageia 3 already ticked off. Not bad going 🙂


Share and Enjoy:
  • Digg
  • StumbleUpon
  • Facebook
  • Yahoo! Buzz
  • Twitter
  • Google Bookmarks
  • Slashdot