Installing Gentoo Linux on ZFS with NVME Drive (Part 1)

Introduction

A few weeks ago, I wrote about how I would be dumping Windows and making the move to Linux as my primary operating system.  I’ve now done it and as I’m writing this my PC is now running Gentoo Linux with ZFS as the operating system for my boot drive and mounted data.  This isn’t a standard configuration for Linux so I thought I’d share some of the basics on how I got it up and working.

The first important things to address are what Gentoo and ZFS actually are.  As you may remember I was wrestling between FreeBSD and Linux but couldn’t find a way to make FreeBSD work (I actually tried it and still couldn’t justify it to myself).  One of the things I really liked about FreeBSD is its packaging system – rather than having pre-compiled packages it’s all actually built from source.  Back when I first used Linux (long before the days of Ubuntu) I really didn’t like package-based distributions and, instead, used to enjoy compiling everything from hand on Slackware.  The FreeBSD approach arguably combines the best of both worlds – packages of source code.  Dependencies are resolved but this is done dynamically depending on the configuration options of the selected packages and it is compiled on your own computer.

Gentoo offers a very similar packaging system based on the Linux kernel.  So, Gentoo is fundamentally a source-based Linux distribution with a FreeBSD-inspired packaging solution.  It’s definitely not one for the novice (since it doesn’t even come with an installer or kernel) but does help you get a much better understanding of how everything pieces together on your system and can give you as much or as little finite control as you desire.

ZFS is a filesystem.  It’s an awesome filesystem.  I’ve written before about the configuration of my NAS at home that uses it and even going back a few years I was discussing ZFS.  It offers a lot of the functionality of hardware RAID – but with none of the drawbacks of software RAID.  It offers encryption, de-duplication, performance, snapshots, 128-bit of capacity addressing, caching and much more.  I think it’s the best filesystem available and so the possibility of bringing it to my desktop excites me.

My Setup and Problems

I built my PC last summer and to say it is over-specced is probably an understatement.  I’ve got 28 core (56 threads), 32GB DDR4 2400 ECC RAM, 3 GPUs (Intel, AMD and NVidia), two soundcards, a 1TB Crucial SATA SSD, 6 gigabit network ports and a 512GB NVME M.2 drive.  There are reasons for all my choices and GPU-wise in particular there is lots I can run through in future videos – but for now the main concern is the NVME drive as this is also my boot drive.

I’m leaving the Windows world completely so all my storage is going to be ZFS.  This is going to not just offer me as many benefits as possible but also means I can snapshot and sync this to my onsite NAS as well as offsite backup.  It all sounds good but there are a number of problems that come as a consequence.  So, let’s see what they are.

Linux does not natively support ZFS.  Yes, that’s definitely a big issue.  There is a Linux ZFS module but due to a differing license (ZFS’ origins were from the commercial world) it cannot be included in the mainline kernel.  This means that it’s never going to be as easy to get up and running as a more common filesystem.  This can be solved by getting the ZFS source code and building the modules against the specific kernel that you are using.  Not too much of a problem but to install Gentoo we need an existing bootable Linux system to start from.  That means we need a Linux boot CD that includes ZFS.  These are more rare, but thankfully the Gentoo ZFS maintainer provides ones of these on his personal site.

Grub doesn’t support ZFS amazingly well.  Out the box it has no ZFS support and although you can compile it with libzfs support there are still some issues it has meaning that you cannot use all the features of ZFS.  The solution here (as well as compiling Grub against libzfs) is to create two separate ZFS “pools”.  You can operate ZFS in “whole disk” mode but for our use-cases we will partition the disk and have one small boot partition just to contain Grub configuration and kernel/initramfs images with the minimum ZFS settings whilst we have the rest of the disk allocated to our large zpool.

Swap is a little bit rubbish on ZFS in terms of performance and there are other issues related to it.  This kind of makes sense though.  One thing ZFS does , in order to kick-ass, is eat a lot of RAM.  You use swap when you run out of RAM.  So using your RAM to back the memory that’s being used to store your RAM that you’ve just run out of seems a little, errm, odd?  I will not be using ZFS for swap for this reason and will be going with a traditional Linux swap partition.

Finally my NVME disk really throws a further spanner in the work as most of the default kernel images I can find do not come with NVME support, further the ZFS initramfs images I’ve seen for Gentoo also do not import zpools on boot correctly from NVME disk.  As such I’m going to have to compile my own kernel with NVME built in (not as a module) and modify the initramfs to support this.

Before I start this I should say that I ended up spending a full day breaking my computer before wiping it and starting over again for a second attempt.  What you see here is learning from the mistakes I already made and there were definite frustrations during this process.  Please try this in a virtual machine before trying it with your real system.

Installing Gentoo Linux on ZFS with an NVME Drive

First I need to give a huge shoutout to Fearedbliss – the Gentoo Linux ZFS maintainer who has an article on the Gentoo wiki page talking through the steps to get this all up and running.  I had to make modifications (and added a few extra bits in) but many of the tools and steps here are thanks to his excellent work.  Please check out his page for his steps.

The Gentoo install is a manual process which follows these high-level steps:

1. Boot another Linux system
2. Setup your disk partitioning
3. Copy core Gentoo files to your new partition
4. Chroot into your Gentoo disk
5. Install a Linux kernel
6. Install Grub and any other core applications you want
7. Reboot

This tutorial assumes you have an EFI-capable machine and will be using it.  If you do not want to use ZFS or do not have an EFI system then there are other tutorials that will be better suited to you.

You need to start by downloading a ZFS-enabled Linux system.  Fearedbliss’ page has a bunch of System Rescue CD links that you can use.  Do not use the standard Gentoo installer as it lacks ZFS support.  When you boot off the CD (or as a memory stick) make sure you boot it in UEFI mode.  If you do not you’ll find that many things are broken later on and you have to start again.  Let me feel the pain so you don’t have to.  You should see a really boring black boot menu rather than one with a pretty blue background if the UEFI one is selected.

The first thing I would recommend once you’ve got to a Linux command prompt is to list all disks visible on your system.

lsblk

Once you’ve run this you should see all disks and their sizes.  If you cannot see all disks you are going to be working on then stop now – don’t delete stuff from disks if you’ve only got half of the system you want.  For me the disk I wanted was the first nvme disk which was named nvme0n1.  Your naming may vary.

The next belt-and-bracers step is to confirm that you have an IP address, your network card was found and that you can ping out to the internet.  You definitely need a network connection fo this so, again, best to check before you start deleting things.

ip addr

ping www.google.com

It’s now time to run parted to edit our partition table.  Note that once you type something in parted it happens immediately and is not staged – so caution is required.  Start it for optimal partition boundaries on the disk you found previously.

parted -a optimal /dev/nvme0n1

Now we’re in here my partition table is going to be split as follows:

#	Usage	Size
1	BIOS Boot Partition	3 MiB
2	EFI Partition	100 MiB
3	ZFS (Boot Pool)	500 MiB
4	Swap	40,000 MiB
5	ZFS (Main Pool)	All remaining

Once parted starts you can get this partition setup by typing in the following commands.  If you want a samller partition size for swap just reduce 40603 for 603 + size of swap in MiB.  I chose 40,000 as I have 32GB RAM and that leaves a few extra GB spare for swap.  If you ever want to hibernate the only real requirement is that you have at least the size of your RAM available for swap.

unit mib
mklabel gpt
mkpart primary 1 3
mkpart primary 3 103
mkpart primary 103 603
mkpart primary 603 40603
mkpart primary 40603 -1
name 1 grub
name 2 esp
name 3 boot
name 4 swap
name 5 rpool
set 1 bios_grub on
set 2 boot on
print
quit

Before parted quit a copy of your partition table should have been printed and it should match what you were expecting.

Now we can setup our EFI partition as FAT32.  Unfortunately no ZFS here due to the EFI specification so we have little choice but to make it FAT32.

mkfs.fat -F32 /dev/nvme0n1p2

We now want to create our two zpools.  The main zpool enables basic compression whilst the boot zpool is created with no additional features (-d).  Both zpools have their cache disabled, default mount points set and are then temporarily mounted in /mnt/gentoo within our system resuce boot CD.  The main pool (called rpool) also has a bunch of filesystems created within it – mounted as the root, home and / directories.

When creating the zpool we’re specifying exactly which features we want.  This is due to a mismatch between ZFS versions on the available installer and within the kernel.  Specifying exactly which features we want reduces this issue.  If you want all ZFS features enables then you need to check your ZFS version in the boot disk is lower than what you’ll be using when you run the kernel.  This will make it easier for you.

zpool create -f -d -o feature@async_destroy=enabled -o feature@empty_bpobj=enabled -o feature@lz4_compress=enabled -o feature@multi_vdev_crash_dump=enabled -o feature@spacemap_histogram=enabled -o feature@enabled_txg=enabled -o feature@hole_birth=enabled -o feature@extensible_dataset=enabled -o feature@embedded_data=enabled -o feature@bookmarks=enabled -o feature@filesystem_limits=enabled -o feature@large_blocks=enabled -o feature@sha512=enabled -o feature@skein=enabled -o feature@edonr=enabled -o feature@userobj_accounting=enabled -o ashift=12 -o cachefile=none -O compression=lz4 -m none -R /mnt/gentoo rpool /dev/nvme0n1p5
zfs create rpool/ROOT
zfs create -o mountpoint=/ rpool/ROOT/gentoo
zfs create -o mountpoint=/home rpool/HOME
zfs create -o mountpoint=/root rpool/HOME/root
zpool create -f -d -o ashift=12 -o cachefile=none -m /boot -R /mnt/gentoo boot /dev/nvme0n1p3

To check that these are configured as you want inspect the zpool and filesystems.

zpool status
zfs list

With ZFS now setup we can turn on and enable swap.

mkswap /dev/nvme0n1p4
swapon /dev/nvme0n1p4

We’re almost ready to go into our Gentoo chroot so first let’s go to the temporary mount directory and create a mount point for the EFI partition then mount the FAT32 we created to this folder.

cd /mnt/gentoo
mkdir boot/efi
mount /dev/nvme0n1p2 boot/efi

With the disks all setup we’re now ready to install the core Gentoo filesystem (known as a Stage 3 tarball).  You can download this however you want – using wget and the URL to get it from Gentoo’s website, from a separate memory stick that you mount or I used SFTP to grab it from another machine on my network.  Once obtained make sure you place it in /mnt/gentoo and then you need to extract it before removing it.

tar xpf stage3.tar.bz2
rm stage3.tar.bz2

There are just a couple of things we need to do in our Gentoo filesystem now.  Ensuring we are still in /mnt/gentoo we want to create a folder for ZFS configuration, copy our DNS settings then rebind /dev, /proc and /sys in to /mnt/gentoo.

mkdir etc/zfs
cp /etc/resolv.conf etc
mount --rbind /dev dev
mount --rbind /proc proc
mount --rbind /sys sys

At this point we’re ready to enter our kernel-less Gentoo system.  We can do this with a chroot and watch our command prompt change.  Now what was /mnt/gentoo is simply /.

env -i HOME=/root TERM=$TERM chroot . bash -l

Since we can now consider any changes we make affecting our Gentoo system directly I started to make some personalisations at this point – starting with the system locale.  We can do some of this now and the rest later once other components have been installed.  In all my examples I will be using nano to edit config files but feel free to use vi.  If you haven’t used nano before you can exit with CTRL+X, press Y to save when prompted then hit enter to save using the same filename.

Start by editing the locale file.

nano -w /etc/locale.gen

In my case I enabled the UK locale by adding a line at the end:

en_GB.UTF-8 UTF-8

You can then regenerate the system locale.

locale-gen

Once that’s done you need to setup your timezone.  I’m in Europe/London so created a symbolic link between the timezone information and system timezone setting files.  You can just look in /usr/share/zoneinfo to see all timezones.

ln -sf /usr/share/zoneinfo/Europe/London /etc/localtime

Once that was done checking my date with “date” showed the wrong time so I needed to set it.  For this example let’s assume it is March 7th at 1.02pm in 2015.

date 030713022015

Next step was to edit the hostname file and specify my computer name.  I like GuyPc so load up the editor:

nano -w /etc/conf.d/hostname

And then set the contents of the file to:

hostname="GuyPc”

The Gentoo partition needs to now be told about all the partitions we’ve setup and to become aware of what is mounted.  I say all – but in reality it should not auto-mount the EFI partition (this can break by creating folders at the wrong time), it is unaware of ZFS and so that pretty much just leaves swap.  Nonetheless we want to setup our mtab file and create a basic fstab.

ln -sf /proc/self/mounts /etc/mtab
nano -w /etc/fstab

The file should then contain:

/dev/nvme0n1p2    /boot/efi       vfat            defaults,noauto        1 2
/dev/nvme0n1p4    none            swap            sw                     0 0

The next step is to setup how Gentoo will build packages for you.  Each CPU has different optimisations and so whilst the basic configuration options will work for most PCs they won’t necessarily optimise media applications as best they could.  The quick-and-easy-but-probably-not-amazing solution I did for this was to get a list of all my CPU flags and store it in a file called flags.

cat /proc/cpuinfo | grep flags | tail --lines=1 > flags

You then want to open up the Gentoo package management configuration file.

nano /etc/portage/make.conf

Find the line CPU_FLAGS_X86 and remove everything in the quotes.  Go to the end of the first quote and read in your flags file (in Nano this can be done with CTRL+R and enter flags as the filename).  All your CPU flags will appears.  Add mmxext at the end and if you have pni in your list replace it with SSE3.  There may be valid reasons you don’t want to do this (for example you may specifically not want both MMX and newer extensions on there) but this is a quick way to get the system up with some optimizations before we install more packages later (gentoolkit  and cpuid2cpuflags can help here -check the Wikifor details).

There are a few other settings you want to make.  If you’re just using keyboard and mouse then you want to specify default input device to be evdev.

INPUT_DEVICES="evdev"

If you have an AMD graphics card you set it to radeon, you can use nouveau for Nvidia cards or both with them being space separated.

VIDEO_CARDS="radeon"

You want to also specify your language locale again here.

LINGUAS="en en_GB"

Finally specify any other settings to pass along to make.  The -j flag specifies how many simultaneous files to compile.  A good rule of thumb is as many cores as you have plus one.  In my case that gives me 57.

MAKEOPTS="-j57"

Next we’re ready to start configuring the package management system.  We take a default configuration file then we’re going to edit it and add in a new overlay.  An overlay is the name for a repository in the Gentoo package management system since it overlays new packages (or different versions of packages) on top of the base system.  This repository is from the ZFS maintainer and provides some useful tools.

cp /usr/share/portage/config/repos.conf /etc/portage/repos.conf
nano -w /etc/portage/repos.conf

Place this at the end of the file.

[bliss-overlay]
location = /var/lib/overlays/bliss-overlay
sync-type = git
sync-uri = https://github.com/fearedbliss/bliss-overlay.git
auto-sync = no

We then want to perform an initial package sync and install Git so that we can sync other package sources.

emerge --sync
emerge dev-vcs/git

Once that’s done edit our repo config file again:

nano -w /etc/portage/repos.conf

but change the last line we added to automatically sync the Fearedbliss repo.

auto-sync = yes

We’re then ready to perform another sync (this time using Git that we installed previously) and to download the Linux kernel sources.

emerge --sync
emerge sys-kernel/gentoo-sources

We can check the version of the kernel we’ve downloaded by seeing where the /usr/src/linux symbolic link points.  Make sure to note this down as we’ll need this later.

ls -l /usr/src/linux

We can then go into our kernel source tree and download Fearedbliss’ default configuration file.

cd /usr/src/linux
wget https://xyinn.org/gentoo/configs/config-4.9.20-FC.01
mv config-4.9.20-FC.01 ./.config

Open up the .config file.

nano -w .config

Search for “CONFIG_NVME_CORE” and change the “=m” to “=y”.  Do the same for the CONFIG_BLK_DEV_NVME.  This will change the kernel to have full NVME support built-in rather than available as a module.

Once this is done you can the kernel UI configuration tool.

make menuconfig

Hit Save as this will enable Gentoo-specific options that are also missing from the config and accept the .config filename then select Exit and go back to the command prompt.

It’s now time to build your own Linux kernel.  If you haven’t done this before then prepare to wait a while.  Depending on your system this can take a very long time.  If you’ve got a small supercomputer like me then expect it to only take about 3 minutes.  Once again replace the -j57 with same value you entered in your Portage configuration earlier.

make -j57

And Breathe…

Since this is a rather lengthy process I’m splitting this tutorial in to two parts.  This is the end of the first part – we’ve built our own kernel, configured a package management system, hand-crafted our partition tables, configured two ZFS pools and a selection of file systems and learned how to manually chroot in Linux.  Not bad for an OS install.

Next time we’ll be finishing off the kernel, constructing our own initramfs that can boot our ZFS disks, installing Grub with ZFS support and booting our system for the first time.

You can read the second part of this guide here.