Installing Gentoo Linux on ZFS with NVME Drive (Part 1)

Introduction

A few weeks ago, I wrote about how I would be dumping Windows and making the move to Linux as my primary operating system.  I’ve now done it and as I’m writing this my PC is now running Gentoo Linux with ZFS as the operating system for my boot drive and mounted data.  This isn’t a standard configuration for Linux so I thought I’d share some of the basics on how I got it up and working.

The first important things to address are what Gentoo and ZFS actually are.  As you may remember I was wrestling between FreeBSD and Linux but couldn’t find a way to make FreeBSD work (I actually tried it and still couldn’t justify it to myself).  One of the things I really liked about FreeBSD is its packaging system – rather than having pre-compiled packages it’s all actually built from source.  Back when I first used Linux (long before the days of Ubuntu) I really didn’t like package-based distributions and, instead, used to enjoy compiling everything from hand on Slackware.  The FreeBSD approach arguably combines the best of both worlds – packages of source code.  Dependencies are resolved but this is done dynamically depending on the configuration options of the selected packages and it is compiled on your own computer.

Gentoo offers a very similar packaging system based on the Linux kernel.  So, Gentoo is fundamentally a source-based Linux distribution with a FreeBSD-inspired packaging solution.  It’s definitely not one for the novice (since it doesn’t even come with an installer or kernel) but does help you get a much better understanding of how everything pieces together on your system and can give you as much or as little finite control as you desire.

ZFS is a filesystem.  It’s an awesome filesystem.  I’ve written before about the configuration of my NAS at home that uses it and even going back a few years I was discussing ZFS.  It offers a lot of the functionality of hardware RAID – but with none of the drawbacks of software RAID.  It offers encryption, de-duplication, performance, snapshots, 128-bit of capacity addressing, caching and much more.  I think it’s the best filesystem available and so the possibility of bringing it to my desktop excites me.

My Setup and Problems

I built my PC last summer and to say it is over-specced is probably an understatement.  I’ve got 28 core (56 threads), 32GB DDR4 2400 ECC RAM, 3 GPUs (Intel, AMD and NVidia), two soundcards, a 1TB Crucial SATA SSD, 6 gigabit network ports and a 512GB NVME M.2 drive.  There are reasons for all my choices and GPU-wise in particular there is lots I can run through in future videos – but for now the main concern is the NVME drive as this is also my boot drive.

I’m leaving the Windows world completely so all my storage is going to be ZFS.  This is going to not just offer me as many benefits as possible but also means I can snapshot and sync this to my onsite NAS as well as offsite backup.  It all sounds good but there are a number of problems that come as a consequence.  So, let’s see what they are.

Linux does not natively support ZFS.  Yes, that’s definitely a big issue.  There is a Linux ZFS module but due to a differing license (ZFS’ origins were from the commercial world) it cannot be included in the mainline kernel.  This means that it’s never going to be as easy to get up and running as a more common filesystem.  This can be solved by getting the ZFS source code and building the modules against the specific kernel that you are using.  Not too much of a problem but to install Gentoo we need an existing bootable Linux system to start from.  That means we need a Linux boot CD that includes ZFS.  These are more rare, but thankfully the Gentoo ZFS maintainer provides ones of these on his personal site.

Grub doesn’t support ZFS amazingly well.  Out the box it has no ZFS support and although you can compile it with libzfs support there are still some issues it has meaning that you cannot use all the features of ZFS.  The solution here (as well as compiling Grub against libzfs) is to create two separate ZFS “pools”.  You can operate ZFS in “whole disk” mode but for our use-cases we will partition the disk and have one small boot partition just to contain Grub configuration and kernel/initramfs images with the minimum ZFS settings whilst we have the rest of the disk allocated to our large zpool.

Swap is a little bit rubbish on ZFS in terms of performance and there are other issues related to it.  This kind of makes sense though.  One thing ZFS does , in order to kick-ass, is eat a lot of RAM.  You use swap when you run out of RAM.  So using your RAM to back the memory that’s being used to store your RAM that you’ve just run out of seems a little, errm, odd?  I will not be using ZFS for swap for this reason and will be going with a traditional Linux swap partition.

Finally my NVME disk really throws a further spanner in the work as most of the default kernel images I can find do not come with NVME support, further the ZFS initramfs images I’ve seen for Gentoo also do not import zpools on boot correctly from NVME disk.  As such I’m going to have to compile my own kernel with NVME built in (not as a module) and modify the initramfs to support this.

Before I start this I should say that I ended up spending a full day breaking my computer before wiping it and starting over again for a second attempt.  What you see here is learning from the mistakes I already made and there were definite frustrations during this process.  Please try this in a virtual machine before trying it with your real system.

Installing Gentoo Linux on ZFS with an NVME Drive

First I need to give a huge shoutout to Fearedbliss – the Gentoo Linux ZFS maintainer who has an article on the Gentoo wiki page talking through the steps to get this all up and running.  I had to make modifications (and added a few extra bits in) but many of the tools and steps here are thanks to his excellent work.  Please check out his page for his steps.

The Gentoo install is a manual process which follows these high-level steps:

  1. Boot another Linux system
  2. Setup your disk partitioning
  3. Copy core Gentoo files to your new partition
  4. Chroot into your Gentoo disk
  5. Install a Linux kernel
  6. Install Grub and any other core applications you want
  7. Reboot

This tutorial assumes you have an EFI-capable machine and will be using it.  If you do not want to use ZFS or do not have an EFI system then there are other tutorials that will be better suited to you.

You need to start by downloading a ZFS-enabled Linux system.  Fearedbliss’ page has a bunch of System Rescue CD links that you can use.  Do not use the standard Gentoo installer as it lacks ZFS support.  When you boot off the CD (or as a memory stick) make sure you boot it in UEFI mode.  If you do not you’ll find that many things are broken later on and you have to start again.  Let me feel the pain so you don’t have to.  You should see a really boring black boot menu rather than one with a pretty blue background if the UEFI one is selected.

The first thing I would recommend once you’ve got to a Linux command prompt is to list all disks visible on your system.

lsblk

Once you’ve run this you should see all disks and their sizes.  If you cannot see all disks you are going to be working on then stop now – don’t delete stuff from disks if you’ve only got half of the system you want.  For me the disk I wanted was the first nvme disk which was named nvme0n1.  Your naming may vary.

The next belt-and-bracers step is to confirm that you have an IP address, your network card was found and that you can ping out to the internet.  You definitely need a network connection fo this so, again, best to check before you start deleting things.

ip addr

ping www.google.com

It’s now time to run parted to edit our partition table.  Note that once you type something in parted it happens immediately and is not staged – so caution is required.  Start it for optimal partition boundaries on the disk you found previously.

parted -a optimal /dev/nvme0n1

Now we’re in here my partition table is going to be split as follows:

# Usage Size
1 BIOS Boot Partition 3 MiB
2 EFI Partition 100 MiB
3 ZFS (Boot Pool) 500 MiB
4 Swap 40,000 MiB
5 ZFS (Main Pool) All remaining

Once parted starts you can get this partition setup by typing in the following commands.  If you want a samller partition size for swap just reduce 40603 for 603 + size of swap in MiB.  I chose 40,000 as I have 32GB RAM and that leaves a few extra GB spare for swap.  If you ever want to hibernate the only real requirement is that you have at least the size of your RAM available for swap.

unit mib
mklabel gpt
mkpart primary 1 3
mkpart primary 3 103
mkpart primary 103 603
mkpart primary 603 40603
mkpart primary 40603 -1
name 1 grub
name 2 esp
name 3 boot
name 4 rpool
name 5 swap
set 1 bios_grub on
set 2 boot on
print
quit

Before parted quit a copy of your partition table should have been printed and it should match what you were expecting.

Now we can setup our EFI partition as FAT32.  Unfortunately no ZFS here due to the EFI specification so we have little choice but to make it FAT32.

mkfs.fat -F32 /dev/nvme0n1p2

We now want to create our two zpools.  The main zpool enables basic compression whilst the boot zpool is created with no additional features (-d).  Both zpools have their cache disabled, default mount points set and are then temporarily mounted in /mnt/gentoo within our system resuce boot CD.  The main pool (called rpool) also has a bunch of filesystems created within it – mounted as the root, home and / directories.

zpool create -f -o ashift=12 -o cachefile=none -O compression=lz4 -m none -R /mnt/gentoo rpool /dev/nvme0n1p5
zpool create -f -d -o ashift=12 -o cachefile=none -m /boot -R /mnt/gentoo boot /dev/nvme0n1p3
zfs create rpool/ROOT
zfs create -o mountpoint=/ rpool/ROOT/gentoo
zfs create -o mountpoint=/home rpool/HOME
zfs create -o mountpoint=/root rpool/HOME/root

To check that these are configured as you want inspect the zpool and filesystems.

zpool status
zfs list

With ZFS now setup we can turn on and enable swap.

mkswap /dev/nvme0n1p4
swapon /dev/nvme0n1p4

We’re almost ready to go into our Gentoo chroot so first let’s go to the temporary mount directory and create a mount point for the EFI partition then mount the FAT32 we created to this folder.

cd /mnt/gentoo
mkdir boot/efi
mount /dev/nvme0n1p2 boot/efi

With the disks all setup we’re now ready to install the core Gentoo filesystem (known as a Stage 3 tarball).  You can download this however you want – using wget and the URL to get it from Gentoo’s website, from a separate memory stick that you mount or I used SFTP to grab it from another machine on my network.  Once obtained make sure you place it in /mnt/gentoo and then you need to extract it before removing it.

tar xpf stage3.tar.bz2
rm stage3.tar.bz2

There are just a couple of things we need to do in our Gentoo filesystem now.  Ensuring we are still in /mnt/gentoo we want to create a folder for ZFS configuration, copy our DNS settings then rebind /dev, /proc and /sys in to /mnt/gentoo.

mkdir etc/zfs
cp /etc/resolv.conf etc
mount --rbind /dev dev
mount --rbind /proc proc
mount --rbind /sys sys

At this point we’re ready to enter our kernel-less Gentoo system.  We can do this with a chroot and watch our command prompt change.  Now what was /mnt/gentoo is simply /.

env -i HOME=/root TERM=$TERM chroot . bash -l

Since we can now consider any changes we make affecting our Gentoo system directly I started to make some personalisations at this point – starting with the system locale.  We can do some of this now and the rest later once other components have been installed.  In all my examples I will be using nano to edit config files but feel free to use vi.  If you haven’t used nano before you can exit with CTRL+X, press Y to save when prompted then hit enter to save using the same filename.

Start by editing the locale file.

nano -w /etc/locale.gen

In my case I enabled the UK locale by adding a line at the end:

en_GB.UTF-8 UTF-8

You can then regenerate the system locale.

locale-gen

Once that’s done you need to setup your timezone.  I’m in Europe/London so created a symbolic link between the timezone information and system timezone setting files.  You can just look in /usr/share/zoneinfo to see all timezones.

ln -sf /usr/share/zoneinfo/Europe/London /etc/localtime

Once that was done checking my date with “date” showed the wrong time so I needed to set it.  For this example let’s assume it is March 7th at 1.02pm in 2015.

date 030713022015

Next step was to edit the hostname file and specify my computer name.  I like GuyPc so load up the editor:

nano -w /etc/conf.d/hostname

And then set the contents of the file to:

hostname="GuyPc”

The Gentoo partition needs to now be told about all the partitions we’ve setup and to become aware of what is mounted.  I say all – but in reality it should not auto-mount the EFI partition (this can break by creating folders at the wrong time), it is unaware of ZFS and so that pretty much just leaves swap.  Nonetheless we want to setup our mtab file and create a basic fstab.

ln -sf /proc/self/mounts /etc/mtab
nano -w /etc/fstab

The file should then contain:

/dev/nvme0n1p2    /boot/efi       vfat            defaults,noauto        1 2
/dev/nvme0n1p5    none            swap            sw                     0 0

The next step is to setup how Gentoo will build packages for you.  Each CPU has different optimisations and so whilst the basic configuration options will work for most PCs they won’t necessarily optimise media applications as best they could.  The quick-and-easy-but-probably-not-amazing solution I did for this was to get a list of all my CPU flags and store it in a file called flags.

cat /proc/cpuinfo | grep flags | tail --lines=1 > flags

You then want to open up the Gentoo package management configuration file.

nano /etc/portage/make.conf

Find the line CPU_FLAGS_X86 and remove everything in the quotes.  Go to the end of the first quote and read in your flags file (in Nano this can be done with CTRL+R and enter flags as the filename).  All your CPU flags will appears.  Add mmxext at the end and if you have pni in your list replace it with SSE3.  There may be valid reasons you don’t want to do this (for example you may specifically not want both MMX and newer extensions on there) but this is a quick way to get the system up with some optimizations before we install more packages later (gentoolkit  and cpuid2cpuflags can help here -check the Wikifor details).

There are a few other settings you want to make.  If you’re just using keyboard and mouse then you want to specify default input device to be evdev.

INPUT_DEVICES="evdev"

If you have an AMD graphics card you set it to radeon, you can use nouveau for Nvidia cards or both with them being space separated.

VIDEO_CARDS="radeon"

You want to also specify your language locale again here.

LINGUAS="en en_GB"

Finally specify any other settings to pass along to make.  The -j flag specifies how many simultaneous files to compile.  A good rule of thumb is as many cores as you have plus one.  In my case that gives me 57.

MAKEOPTS="-j57"

Next we’re ready to start configuring the package management system.  We take a default configuration file then we’re going to edit it and add in a new overlay.  An overlay is the name for a repository in the Gentoo package management system since it overlays new packages (or different versions of packages) on top of the base system.  This repository is from the ZFS maintainer and provides some useful tools.

cp /usr/share/portage/config/repos.conf /etc/portage/repos.conf
nano -w /etc/portage/repos.conf

Place this at the end of the file.

[bliss-overlay]
location = /var/lib/overlays/bliss-overlay
sync-type = git
sync-uri = https://github.com/fearedbliss/bliss-overlay.git
auto-sync = no

We then want to perform an initial package sync and install Git so that we can sync other package sources.

emerge --sync
emerge dev-vcs/git

Once that’s done edit our repo config file again:

nano -w /etc/portage/repos.conf

but change the last line we added to automatically sync the Fearedbliss repo.

auto-sync = yes

We’re then ready to perform another sync (this time using Git that we installed previously) and to download the Linux kernel sources.

emerge --sync
emerge sys-kernel/gentoo-sources

We can check the version of the kernel we’ve downloaded by seeing where the /usr/src/linux symbolic link points.  Make sure to note this down as we’ll need this later.

ls -l /usr/src/linux

We can then go into our kernel source tree and download Fearedbliss’ default configuration file.

cd /usr/src/linux
wget https://xyinn.org/gentoo/configs/config-4.9.20-FC.01
move config-4.9.20-FC.01 ./.config

Open up the .config file.

nano -w .config

Search for “CONFIG_NVME_CORE” and change the “=m” to “=y”.  Do the same for the CONFIG_BLK_DEV_NVME.  This will change the kernel to have full NVME support built-in rather than available as a module.

Once this is done you can the kernel UI configuration tool.

make menuconfig

Hit Save as this will enable Gentoo-specific options that are also missing from the config and accept the .config filename then select Exit and go back to the command prompt.

It’s now time to build your own Linux kernel.  If you haven’t done this before then prepare to wait a while.  Depending on your system this can take a very long time.  If you’ve got a small supercomputer like me then expect it to only take about 3 minutes.  Once again replace the -j57 with same value you entered in your Portage configuration earlier.

make -j57

And Breathe…

Since this is a rather lengthy process I’m splitting this tutorial in to two parts.  This is the end of the first part – we’ve built our own kernel, configured a package management system, hand-crafted our partition tables, configured two ZFS pools and a selection of file systems and learned how to manually chroot in Linux.  Not bad for an OS install.

Next time we’ll be finishing off the kernel, constructing our own initramfs that can boot our ZFS disks, installing Grub with ZFS support and booting our system for the first time.

You can read the second part of this guide here.

Advertisements

3 thoughts on “Installing Gentoo Linux on ZFS with NVME Drive (Part 1)

  1. Dear Guy, Robot,

    May I start by saying just how useful these two parts you’ve so caringly assembled are. Amazing!!!

    I’m a newbie at Linux but installed Arch on the machine below a year ago on my own through here and there research. After messing around with virtualbox it messed up my system saying some kernel modules were missing. I installed Arch on vb within Arch so as to get to learn more but eventually it died. I did have Gnome, KDE Plasma, Cinnamon and Mate installed. I was just messing around to see where it would take me. Updates were my enemy.

    I bought a dell latitude E6230 refurbished (for wrecking, testing and learning Linux), i5 duo and upgraded to 8G Ram. I’ve just dbanned my hard drive and am going for Gentoo, (Lord have mercy!). As I read, watch and generally research stuff, everything is so vague and blurred,

    I want to use Gentoo specifically for pro-audio music production as it will not bloat and for low-latency. I’ve read bits and bobs, (due to time restraints, I’m a Maths teacher in an inner city secondary school in London and I too had a major life-changing event – nearly lost my dad, separation (my ex will not even let me speak to my 3 daughters – took them to Greece and am now going through courts), to mention a few, so I would like to think that I know “slightly”, where you’re coming from), such as; the Gentoo manual, Project sound-how to enable real time for multimedia apps, Kernel configuration guide, complete handbook on configuring the system and videos.

    I’m 50, an Electronics & Telecommunications Engineering Masters graduate (graduated 1999 but don’t let that fool you!) and my passion has always been music.

    Everything is so confusing what with hdd’s and ssd’s needing setting up. vfat and mklabel dos, and do i use overlays for low-latency or the sound project or the gentoo studio project (which I think is outdated as I did try it out and even configured my own ow-latency kernel from that but could not load xfce after that so I did a Gutemann dban that took 36 hours)

    I’ve partitioned both with cfdisk and parted and that was not too bad. I understand basics and what they do such as mkfs and mounting and have the linux command line book, but man it’s a minefield out there.

    This is the first time I’ve seen/read such a beautifully laid out piece of documentation, My congratulations go out to you sir!

    I will be buying a 3xs system from scan computers with two ssd’s. I will be storing music files on a 3TB external HDD. I’m thinking of running Gentoo on one ssd and windows on the other until I get to grips with it. In the meantime, I am going to install Gentoo on my dell with 320G hdd and will be following your tutorials with my spec obviously. IF, you have the time and can get back to me with a little advice, God bless you. If not, God bless you. Either way you’re a star.

    Apologies for going on, I would just like to say do not be phased by anyone or anything and go get them there robots. You have found the meaning of life my friend and I wish you all the best, health, happiness and every success!!!

    Kind regards

    Kyri

    Like

    1. Hi Kyri,

      Thanks for your kind words and leaving a message. It’s always great to hear from people and to know that anything I create is useful to others 🙂 A 3xs from Scan will certainly be powerful enough for you but depending on what you want to get out of it Gentoo may not be the way to go. I’m a great believer that diving in at the deep end (Gentoo is pretty close to the deep end although certainly not all the way in) is a good way to learn about things from top to bottom (having built your own kernel it does sound like you’ve already been there to some extent) so I’d guess it depends whether you are aiming for productivity with music production or a fun journey to get there. If you don’t mind taking several detours learning about Linux en-route then Gentoo is a great option but if you want to make the most out of your system then even something like Fedora, Debian or Ubuntu may well be better suited to you – there’s no reason with any of those you can’t still compile things from source or have your own kernel if that is what makes most sense to you. There’s also something like AV Linux for something more niche.

      When it comes to music production this is far from my speciality so the specifics there’s not a huge amount I can advise you on (apart from buy SSDs for quick access and have a decent amount of additional storage – all of which you’ve taken care of). The only thing I would say is that from my experience so far in 2017 of trying to do video and audio editing workflow for my YouTube channel in Linux it’s a lot more difficult than other platforms. In the video world there is nothing open source/free that is remotely comparable to the tools available in Windows or Mac OS. For sound recording and very basic tweaking I use Audacity in Linux which works fine for me – I suspect your needs may be greater. I don’t know if the audio experience is similar to the video one but if so prepare to be somewhat frustrated and underwhelmed. I suspect there’s a reason that many studios still use Macs – you’ve always got the option of going for a home-brew Mac for your audio purposes. I may dislike current Apple hardware but a Hackintosh designed for audio could be something really fun that offers you a much easier and better end-experience out of it with some tech learning still in there as well. Don’t want to dissuade you from Linux but I do think for any technological choice there needs to be a reason for in – mine for Gentoo and Linux is primarily the fun, carry on with Linux if that appeals to you more than getting into your pro-audio 🙂 I do think MacOS will probably tick a lot of the requirements you have – but the software may well be expensive.

      If you are carrying on down the Gentoo road I’m going to have some more general purpose videos (one for XFCE actually pretty soon) and I’m happy to take requests for specifics anybody is interested in. I did also have a quick look around for you and it does look like the Gentoo Studio distribution is still maintained (the author is still posting about it https://forums.gentoo.org/viewtopic-t-931180-postdays-0-postorder-asc-start-275.html?sid=a23ddf70b5568aec63db36d793ae6b06 and seems to help with support via IRC) and if you haven’t seen it this may be worth a look too http://www.bandshed.net/avlinux/

      Thanks again for your comment and let me know how it goes!

      Guy

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s