Installing Gentoo Linux on ZFS with NVME Drive (Part 2)

Ready for More?

If you missed it my last post was looking at how to get Gentoo Linux installed on a ZFS filesystem with NVME disks.  Due to the size of this tutorial I’ve split it into two parts and if you’re here you should definitely have read the first part of this tutorial for any of the rest of this to make sense.

Last time we got our disks setup, installed the core Gentoo system, chrooted into a kernel-free Gentoo system and built a bespoke kernel.  Today we’re going to install our kernel, setup Grub, finish configuring our system and boot it for the first time straight from ZFS.

Back to the Kernel

Last time we’d build the kernel but not done anything else with it.  It’s time to copy it to your /boot folder and to install the modules to your system.

make modules_install
make install

Your kernel has now been copied to /boot ready for us to use later on.

We’re now ready to get everything setup for Grub.  We first need to edit the Portage configuration and tell it that we’re using EFI-64 Grub.

nano -w /etc/portage/make.conf

Add the line.

GRUB_PLATFORMS="efi-64"

Next we want to edit the list of which packages we accept.

nano -w /etc/portage/package.accept_keywords

We add in four packages that are defined as “in testing” rather than stable.  Normally testing packages would be filtered but we want the latest-and-greatest of these to enable ZFS support.  The “~” is the “in-testing” indicator and this file expressly includes packages normally ignored.

sys-kernel/spl ~amd64
sys-fs/zfs ~amd64
sys-fs/zfs-kmod ~amd64
sys-boot/grub ~amd64

We can then edit the default configuration for how we will build Grub:

nano -w /etc/portage/package.use/grub

And we add a line saying that we want to compile it against libzfs.

sys-boot/grub libzfs

Once we’ve got those settings done we can pull down the tools to create a ZFS initramfs and Grub itself.  Grub will also pull down the ZFS kernel modules and other utilities.

emerge bliss-initramfs grub

Depending on how old your Stage 3 tarball is you may also want to update all packages already installed:

emerge -uDNav @world

Next we’re ready to install Grub.  First run some grub-probe commands to ensure that “ZFS” is returned.  If it is not something is wrong – do not carry on.

mkdir /build-initramfs
cd /build-initramfs
grub-probe /boot
grub-probe /boot/efi

Now we’re ready to install Grub.  We also remount the EFIVars partition here as rw.  If you get any warnings about EFIVars (or anything else for that matter) then something is wrong – do not carry on.

mount -o remount,rw /sys/firmware/efi/efivars/
grub-install --efi-directory=/boot/efi /dev/nvme0n1

We can then open up our Grub configuration file.

nano -w /boot/grub/grub.cfg

This will be empty so we can supply a default set of values.  Make sure to change the kernel version number here from 4.9.16-gentoo-FC.01 to whatever version you have.  You can confirm this by looking in /lib/modules.

 set timeout=1
 set default=0
 insmod part_gpt
 insmod fat
 insmod efi_gop
 insmod efi_uga
 menuentry "Gentoo" {
   linux /@/kernels/4.9.16-gentoo-FC.01/vmlinuz root=rpool/ROOT/gentoo by=id elevator=noop quiet logo.nologo
   initrd /@/kernels/4.9.16-gentoo-FC.01/initrd
}

We’re now ready to create an initial ramfs for ZFS that we’ll modify to support nvme.  Run the blizz-initramfs tool.

bliss-initramfs

When prompted select option “1“ for ZFS, “n” as you don’t want to use the suggested kernel and then enter your kernel version (for example 4.9.16-gentoo-FC.01).

We can now create the path for our kernel and initramfs to be stored in, copy the initramfs and then move the kernel and its related files to this folder:

mkdir -p /boot/kernels/4.9.16-gentoo-FC.01
mv initrd-4.9.16-gentoo-FC.01 /boot/kernels/4.9.16-gentoo-FC.01/initrd
cd /boot
mv config-4.9.16-gentoo-FC.01 kernels/4.9.16-gentoo-FC.01/config
mv vmlinuz-4.9.16-gentoo-FC.01 kernels/4.9.16-gentoo-FC.01/vmlinuz
mv System.map-4.9.16-gentoo-FC.01 kernels/4.9.16-gentoo-FC.01/System.map

We now need to modify the initramfs.  This is where it can get a little tricky.  We start by going back to our home directory and then open up a new file called extract.sh.

cd /root
nano -w extract.sh

The contents of this file, as shown below, take a copy of the initrd we just created and extract it to a folder called “ird” in root’s home directory.  This allows us to edit it.

mkdir /root/ird 2>/dev/null
rm -Rf /root/ird/*
cd /root/ird
zcat /boot/kernels/4.9.16-gentoo-FC.01/initrd |cpio -idmv

We then open make.sh:

nano -w make.sh

And use the following contents to re-create our initrd and copy it back to /boot.

cd /root/ird
find . -print0 |cpio --null -ov --format=newc |gzip -9 >  /boot/kernels/4.9.16-gentoo-FC.01/initrd

Finally we set both of these to be executable and then extract the initrd

bash ./extract.sh

You will now be in /root/ird and can edit the initial ramdisk.  We need to turn off the default ZFS-mounting behaviour and expressly mount our zpools.  Open up the init file.

nano -w init

Then go to the very end of the file (CTRL+W, CTRL+V in nano) and change these three lines:

CheckAndRunTriggers
MountRoot
MountUsrIfNeeded

To these two lines:

zpool import rpool -R /mnt/root
zpool import boot -R /mnt/root

If you have additional ZFS pools you have created now, or in the future, you may as well add them in here too.

Once you’ve saved your init file update the initrd.

bash /root/make.sh

Now you have a custom initramfs that should mount your ZFS pools even on your nvme disk.

Before we reboot we want to enable some ZFS services and disable hardware clock (since we’re using local time).

rc-update add zfs-mount boot
rc-update add zfs-share default
rc-update add zfs-zed default
rc-update delete hwclock boot

Then we can set a password for root.

passwd

We finish off some of our locale settings – first by looking at the locale list, seeing which number matches our requirements and then passing this into locale set before updating our environment settings.  We can then confirm this was a success by checking the locale.

eselect locale list
eselect locale set 3
env-update && source /etc/profile
locale

It’s then time to edit your system keymap.

nano -w /etc/conf.d/keymaps

In my case I just wanted to change the first conf setting

keymap="uk”

Due to the changes I’ve made to my initramfs and potential issues I saw arrise in other testing I have disabled parallel start-up on my system and enabled interactive boot mode so that I can stop certain services from starting.  I’d recommend you do the same by editing your OpenRC config.

nano -w /etc/rc.conf

Then ensuring these two lines have the following settings and are not commented out:

rc_parallel="NO”
rc_interactive="YES”

If you’re using an ATI card you’re going to need your GPU firmware which can be easily installed:

emerge sys-kernel/linux-firmware

We’re then ready to specify which network card we want to use in Gentoo.  You can confirm this again by checking what is already in use:

ip addr

In my case this is enp6s0.  For now I just want to tell the system to enable it and use DHCP at start-up which is as simple as including the network card in the startup scripts.

cd /etc/init.d
ln -s net.lo net.enp6s0
rc-update add net.enp6s0 default

There are then a couple of settings to make to specify my domain name.  I can edit the network configuration file.

nano -w /etc/conf.d/net

And specify my local domain name.

dns_domain_lo="guytp.org"
dns_domain_enp6s0="guytp.org"

I can then also edit my hosts file.

nano -w /etc/hosts

Here I want to put my computer name (GuyPc) at the end of the 127.0.0.1 and ::1 lines.

127.0.0.1    localhost GuyPc
::1          localhost GuyPc

The very last step is to modify the clock configuration.

nano -w /etc/conf.d/hwclock

Here I want to reaffirm I am using local time.

clock="local"

At this point it is time to exit the chroot and reboot after we delete our temporary initramfs folder.

rm -Rf /zpoolbuild-initramfs
exit
reboot

If everything goes to plan you will very shortly be at your Gentoo login prompt where you can log back in as root.

Now it’s time to add a few system tools you’ll want – I have added sudo (so I do not have to login as root), htop (for tracking how everything is running), sysklogd with lograte (as the most simple syslog setup), cronie (for running scheduled tasks), mlocate (for helping me find missing files) and a couple of network utilities I will inevitably need at some future point.  I also enable SSH.  I haven’t run through the SSHD or Sudo configuration here as that’s up to your own preference.

emerge sudo htop gentoolkit app-admin/sysklogd  app-admin/logrotate sys-process/cronie sys-apps/mlocate net-ftp/ftp net-misc/telnet-bsd
rc-update add sysklogd default
rc-update add cronie default
crontab /etc/crontab
rc-update add sshd default

I then create myself a local user with access to sudo and some physical devices and set myself as password.

useradd -m -G users,wheel,audio,cdrom,cdrw,usb,video -s /bin/bash guytp
passwd guytp

The very final step is to snapshot all of this so that I have a known state I can return to at a future date.

zfs snapshot rpool/ROOT/gentoo@2017-04-11-0000-01-INSTALL
zfs snapshot rpool/HOME@2017-04-11-0000-01-INSTALL
zfs snapshot boot@2014-04-11-0000-01-INSTALL

We’re Done!

Quite the install.  It’s certainly not just an Ubuntu next, next, next, done but it is a lot more interesting.  You’ll have learned a lot about how your system is setup, have already discovered where most of the system configuration is and even hand-crafted some start-up scripts to run before your filesystem is even mounted.  That’s a lot more fun than a day in the land of many Linux users.

I had a tonne of problems before I wrote this guide – ranging from not booting into EFI mode to start with and then trying to recover it (oops) to issues with NVME that we’ve addressed here and my network card failing to work within Gentoo (a firmware update was required from Intel).

I’m now on a fully working Linux system (and I’m writing this up in Abiword) but these first steps with ZFS were fundamental to my understanding of the Gentoo package management and ethos and I thoroughly enjoyed two days of frustration to get here.  I’m going to have a separate video documenting my tribulations getting X working and will probably do one in the future when I move over to Wayland as well.

For now I hope this was helpful and let me know how it went for you.