ZFS Installation on Linux (ubuntu)

ZFS or the Zettabyte File System is an extreme robust, ground up re-thinking of how a filesystem should work.  It’s been designed from the ground up to prevent data loss and corruption; in addition to this it offers complete file system management, oppose to just being another layer on top of physical hardware.  This also includes built-in support for SMB/CIFS and NFS file sharing.

ZFS works much differently than most previous file systems created.  Once this file system was opensourced a few linux groups attempted ports for it to be used in POSIX systems.  Due to licensing issues with ZFS and the linux Kernel, there are two primary ways of using ZFS on Linux:

1. A user-space implementation (pre-packaged binary)
2. A kernel-space implementation (manual compilation and installation)

This will detail how to install and use the basic ZFS commands on ubuntu lucid (10.04) server with a RAIDz (RAID-5) implementation example.

ZFS Installation user-space

In ubuntu, the fastest way to get ZFS support is to install the user-space module.  This is a great and easy way to play around with ZFS without getting your hands too dirty.  I’d recommend installing ZFS in kernel-space if you want to do any serious data storage.  Just apt-get install ZFS FUSE:

sudo apt-get install zfs-fuse

… and you are ready to roll on to setting up ZFS

kernel-space

To build ZFS support into the kernel, we need to custom build a version of ZFS from source, to get around Kernel licensing / shipping issues.  We’ll need to add the PPA (Personal Package Archive) for ZFS:

sudo add-apt-repository ppa:dajhorn/zfs

sudo apt-get update

sudo apt-get install ubuntu-zfs

… Just sit back and watch as the ZFS source is downloaded, compiled and installed on your system.  You’re now ready to play with ZFS.

ZFS Drive Configuration 1) Identify disks by path

First, we want to make sure our drives always appear in /dev in the same locations, so we can consistently identify the same drive on linux.  Because of how drives spin up, and are added as they are ready asynchronously, drives will be labeled in a seemingly random order.  This fixes that problem and creates a device node specifically for ZFS disks.

First, we want to find all of our disks using the full PCI bus path:

ls -l /dev/disk/by-path

2) Update /etc/zfs/zdev.conf to map disks by path to zpool

Now we’ll create/edit /etc/zfs/zdev.conf configuration file, to automagically create the zpool disks on boot, in the following format:

<label> <PCI bus location> <# optional note; I use HD serials>

sudo vim /etc/zfs/zdev.conf

disk1 pci-0000:00:1f.2-scsi-1:0:0:0 #HD: 22d3d234-s3a1

…

disk5 pci-0000:00:1f.2-scsi-5:0:0:0 #HD: 2243d634-s3a7

3) Refresh /dev with new device nodes

Now just trigger a udev update to population the new device nodes in dev:

sudo udevadm trigger

ls -l /dev/disk/zpool

disk1

disk2

disk3

disk4

disk5

4) Create GPT partitions for 2TB+ support

Now we will initialize the Drives with a GPT partition, to allow 2TB+ support per disk.  Since we’ve programmatically created the zpool, this is easy to do with a quick bash script:

for x in `seq 5` do;

  sudo parted /dev/disk/zpool/disk${x} mklabel gpt quit;

done

… and now we are ready to create our first ZFS RAIDz zpool!

5) Enable ZFS mounts at boot, and un-mounts at shutdown

ZFS by default on ubuntu will not be automatically mounted or unmounted during startup or shutdown.  To enable this filesystem to be automatically mounted and unmounted, edit /etc/default/zfs, and enable these options:

sudo vim /etc/default/zfs

# Automatically run `zfs mount -a` at system startup if set non-empty.

ZFS_MOUNT=’yes’

#Autmatically run ‘zfs unmount -a` at system shutdown if set non-empty.

ZFS_UNMOUNT=’yes’

Creating RAIDz zpool

A zpool is a pool of disks for use by the ZFS filesystem.  In classical setups, this would be where you would use hardware or software RAID to make a container for your filesystem.  ZFS does this for you.  In our example, we’re going to make a 5 disk RAIDz pool, with autoexpansion enabled (automatically grow pool on drive upgrades).  The amazing thing with ZFS is that as soon as the pool is created, you can start using the filesystem.  No need for a long allocation process, as required in EXT3/4 filesystems.

1) Create zpool

We create a zpool enabling auto expansion, labeling the pool ‘uber’ and using our programmatically created disk ID’s for the disks, automatically mounting the pool to /data when it is created:

sudo zpool create -o autoexpand=on -m /data uber raidz /dev/disk/zpool/disk{1..5} 

2) Check zpools on system (optional)

Once this is done, you can list the current zpools with list command:

sudo zpool list

NAME      SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT

uber       13.6T  10.0T  3.58T    73%  1.00x  ONLINE  -

3) Check zpool status (optional)

And you can check the current status of the zpool with the status command:

sudo zpool status uber

4) Disable built-in SMB/CIFS sharing and NFS sharing

ZFS has built in support for SMB/CIFS and NFS sharing.  I still like to create my own NFS shares however, and therefore, I disable this functionality on my zpool.  To disable these attributes, just run the following commands:

sudo zfs set sharenfs=off uber

sudo zfs set sharesmb=off uber

There are a ton of options that you can set per zpool.  You should check out all the options and set ones that are best for you.  To get a list of all the options, just run:

sudo zfs get all uber

As you can tell, there are a ton of things that this filesystem can do.

Scrubbing zpool

Scrubbing a ZFS file system run through the filesystem and attempts to detect errors on the stored data.  This is done by verifying hashes on the files, and automatically replacing a bad file with a backup copy, if one exists.  For cheap hardware, this should be done weekly, but in practical experience, this should be done monthly.  To kick off a ZFS scrub (which still keeps your filesystem mounted and accessible) just run the following command:

sudo zpool scrub uber

This runs pretty quickly on current hardware.  You can check the status at anytime by running a status check:

sudo zpool status uber

This is really only an introduction to an extremely flexible and powerful filesystem.  It became my primary filesystem almost overnight.

PCBSDからFreeBSDをzfsでインストール

インストールのSSとってない。

ホスト名つけて
新規インストールで、FreeBSDを選択して（PCBSDじゃなくて）

パーティションでは自動設定の
Use Entire Disk （１つのディスクとしてつかう）チェック
FileSystemをZFSに
GPTのパーティションとしてうんぬん。をチェック

気が向いたらSS撮り直す。

rootのパスワードつけて、なんかユーザ作って
オプションなんもいれない。

インストールは結構速い

インストール後、上の方にDHCPのIPアドレス出てるんで覚えといて（でてなきゃあとでifconfigでもすりゃいい）
ログインして、rootになって

キーボードマップの変更（viで記号のマッピングを確認してみる。）
sysconfigいじるのがいいらしいがひとまずコマンドで切り替える

# kbdcontrol -l /usr/share/syscons/keymaps/jp.106.kbd

;; sysconfigは
;; keymap=jp.106を追記してリブート

rc.dをいろいろいじる。
DHCPが動いてる環境ならDHCP設定になってるので、
そこは放置しつつ、

# nameserver [ネームサーバが別途あるならDHCPならたぶんいらん]
sshd_enable=”YES”

を追記
# /etc/rc.d/sshd start
開始

メールのaliases.dbがないなら
# newaliases

時刻確認
# date
Wed Nov 9 01:38:09 UTC 2011

UTCなんでタイムゾーン設定する

# touch /etc/wall_cmos_clock
# ln -s /usr/share/zoneinfo/Asia/Tokyo localtime
;; この2行がイマイチわからん

時刻合わせとNTP起動
# ntpdate ntp.nict.jp

# vi /etc/ntpd.conf
server ntp.nict.jp
:wq

# vi /etc/rc.conf
ntpd_enable=”YES”
:wq

システム更新
# freebsd-update fetch
# freebsd-update install
# reboot
;; でもフェッチでミラーみつからないと失敗した
;; STABLEだと更新できないとか？？RELEASEじゃないとだめ？

とりま今日はここまで

# /etc/rc.d/ntpd start

Serving: The OS.

The Requirements.

The basic requirements of this server haven’t changed much since the first post I made about it months ago, except that I’m no longer living with Pete (the old housemate) and therefore have less data to deal with. Ideally the server needs to do the following:

• Support RAID5 or some other similar redundant storage pooling.
• Act as a SMB/CIFS file server.
• Be capable of running headless, and have adequate remote admin support.
• Act as a torrent box with remote administration support.
• Act as a UPNP Media server/transcoder to server media up to compatible devices (mainly PS3)
• Share music library to iTunes
• Automatically spin down disks when they’re not in use to save power.
• Be stable as a F***ING ROCK.

The Contenders

There were three main options for the OS in my mind when I came back to this project, they were:

• Windows 7: It’s very easy to use, has fantastic hardware support and should be able to satisfy all of the requirements above (and I have a spare licence). The only worry I had was that it is very bulky compared to other OS’s and may impact the power bill over time due to the extra CPU cycles.
• Ubuntu-Server: More complex to use and initially set up but lighter on power and hopefully more stable in the long run. Good well documented guides and support for all of the above requirements.
• Solaris/OpenIndiana: Fantastic support for large file systems through ZFS, but I’ve never use it before and there may be issues with media serving/transcoding. Quite good documentation on all of the above requirements, but steep learning curve.

So, in the interests of a quick setup, I chose first to install Windows 7 Professional. Windows picked up all my hardware right off the bat without any issues and I was installed and running within minutes, BUT when I went to create a new raid array in disk management I discovered that Windows 7 does not support software RAID5, the option is there in the Disk Management console but it’s greyed out.

So with Windows 7 inexplicably ruled out (Seriously Microsoft? Software RAID5 only on server systems? The same systems most likely to use hardware RAID?) my next port of call was Ubuntu linux server edition. So far Ubuntu has been great, I’ve had plenty of issues setting up the drives which I’ll talk about in depth in another post, but after discovering that ubuntu now supports ZFS there is no need to consider OpenIndiana as an option at all.

So it’s Ubuntu… but which version?

Initially I chose to use Ubuntu Server x64 10.04 LTS because I figured it would be better in the long run because it’s an LTS release and is more likely to be bug free. But after messing around with the RAID5 setup through mdadm and then discovering that Linux has ZFS support I chose to upgrade to 11.04

Anecdotally I’ve noticed that building a RAID5 array in 11.04 is about 15-20% slower than it was in 10.04 but formatting EXT4 is now lightning fast where it would previously take minutes to format a single 2TB disk. Additionally 11.04 has newer versions of mdadm and gdisk (absolutely essential if you want to use GPT) in the repositories so there’s no need to build them from source.