Creating a jail with FiFo and Digital Oceans

This tutorial covers a simple setup of FiFo inside of Digital Oceans, starting with the installation continuing with configuration and finally creating our first jail.

Disclaimer

Before you get too excited, I want to put the bad news in the beginning DO’s private networks do not allow for multicast traffic which means FiFo’s node discovery does not work beyond a single node by default. There are a few ways around that (like tunnels or VPN’s), but that’d be beyond the scope of this little write-up.

Now the good news, FiFo works flawlessly on Digital Oceans FreeBSD image, the installation takes just a few minutes, and it works stable. The whole process takes a bit less than 30 minutes, most of that time spent compiling the kernel to enable RCTL and VNET which does not come enabled by default.

The tutorial is not meant to create a production system, security concerns are woefully ignored. This is to spend a few hours playing with FiFo and then tearing it down again. If you plan to run FiFo in production please follow the official documentation or contact us for commercial support.

Creating the Droplet

For this example, we chose the FreeBSD 11.1 ZFS image and the 8GB / 4 CPU droplet so compiling doesn’t take too much time.

Select your location (I like Frankfurt it’s nice and close), and the SSH key to use. Because this will be a single node system there is no need for the “Private networking” option. With all the desired settings selected go ahead and create your droplet. It’ll take a minute or two to boot.

Configuring the system

Now we just install FiFo as covered in the documentation.

We download the kernel code. The command is:

fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/`freebsd-version -k`/src.txz -o /tmp/src.txz
tar -C / -xzf /tmp/src.txz

This will take a moment or two, there are lots of files to download. Once we’re done, we’ll make our own kernel with VNET and RCTL, this too is described in the documentation, but for brevity here we go. We start with creating a custom configuration.

cd /usr/src/sys/amd64/conf
cat > FIFOKERNEL <<EOL
include GENERIC
ident FIFOKERNEL

options         VIMAGE # VNET/Vimage support
options         RACCT  # Resource containers
options         RCTL   # same as above
EOL

cd /usr/src

After we’ve set up the configuration, we build and then install the kernel. Again, this commands will take a few minutes so grab a coffee while you wait.

# Build the kernel
make -j4 buildkernel KERNCONF=FIFOKERNEL
# Install the new kernel
make -j4 installkernel KERNCONF=FIFOKERNEL

With the kernel installed, we set up a few modules to be loaded on the next boot. Namely the Linux emulator the and related modules as well as well as enabling RCTL. After that, we reboot our droplet.

cat <<EOF >> /boot/loader.conf
linux64_load="YES"
linux_load="YES"
fdescfs_load="YES"
linprocfs_load="YES"
linsysfs_load="YES"
tmpfs_load="YES"
mac_portacl_load="YES"
kern.racct.enable=1
EOF

reboot

Once the system is back up, we can double check that our kernel was loaded.

uname -a
FreeBSD fifo-bsd 11.1-RELEASE-p1 FreeBSD 11.1-RELEASE-p1 #0 r324521: Wed Oct 11 09:12:27 UTC 2017     root@fifo-bsd:/usr/obj/usr/src/sys/FIFOKERNEL  amd64

All good, next up we create a few ZFS datasets to organize jails and FiFo data.

zfs create zroot/data
zfs set mountpoint=/data zroot/data
zfs create zroot/data/sniffle
zfs create zroot/data/snarl
zfs create zroot/data/howl
zfs create zroot/jails
zfs set mountpoint=/zroot/jails zroot/jails

* Now is a good time to make a snapshot using the DO dashboard if you plan on going through the setup more than once.

Configuring Networking

We need to configure some networking. FiFo uses a bridge to attach VNETS, we can create it using:

cat <<EOF >> /etc/rc.conf
cloned_interfaces="bridge0"
autobridge_interfaces="bridge0"
autobridge_bridge0="vtnet0"
ifconfig_bridge0="inet 192.168.1.1/24"
EOF

For Jails to have real internet access you’ll need a NAT set up, we will make a simple NAT with PF but this should only be used for testing. Also of course this only allows traffic out, in a production setup an ip pool for jails is best practice.

cat <<EOF > /etc/pf.conf
IP_PUB="`ifconfig vtnet0 | grep "inet " | grep -v "inet 10" | cut -d: -f2 | cut -d\  -f2`"
NET_JAIL="192.168.1.0/24"
scrub in all
nat pass on vtnet0 from \$NET_JAIL to any -> \$IP_PUB
pass out log on vtnet0 proto { tcp, udp, icmp } all
EOF

cat <<EOF >> /etc/rc.conf
pf_enable="YES"
gateway_enable="YES"
EOF

/etc/netstart 
pfctl -nf /etc/pf.conf
service pf start

Installing the base components

Project-FiFo provides a pkg repository so installing the components is quite easy. To get the packages, we first tell the system about our new repository.

mkdir -p /usr/local/etc/pkg/repos
cat <<EOF > /usr/local/etc/pkg/repos/ProjectFiFo.conf
ProjectFiFo: {
  url: "pkg+https://freebsd.project-fifo.net/rel/amd64/11.0",
  mirror_type: "srv",
  enabled: yes
}
EOF
pkg update

Now we install the base services, those are required on every hypervisor not only managing systems. Since we run all on one host, it doesn’t make a huge difference, but let’s do it correctly.

pkg install vmadm chunter zlogin

cat  <<EOF >> /etc/rc.conf
zlogin_enable="YES"
chunter_enable="YES"
vmadm_enable="YES"
EOF

Since the DO FreeBSD comes with an unusual configuration of two IP’s on the same interface, FiFo can’t automatically detect the network configuration. So before we start the services we need to make sure chunter’s config file exists. Since we want everything on one host it’s rather easy we can just copy over the example.

cp /usr/local/lib/chunter/etc/chunter.conf.example /usr/local/lib/chunter/etc/chunter.conf

Now we start the services.

service zlogin start
service chunter start

Installing the management system

After the base services are installed and started we install the management system, this too is done via packages.

pkg install bash fifo-sniffle fifo-snarl fifo-howl fifo-cerberus

Before we start the services, we configure howl to run on 8080 and 8443, so we do not require privileged ports. We do this by editing /data/howl/etc/howl.conf and changing the following lines:

## The port howl listens on for websockets
##
## Default: 80
##
## Acceptable values:
##   - an integer
http_port = 8080

## The port howl listens to.
##
## Default: 443
##
## Acceptable values:
##   - an integer
ssl.port = 8443

Now we can boot up the system, the start order doesn’t matter as the components are self-organizing.

cat  <<EOF >> /etc/rc.conf
sniffle_enable="YES"
snarl_enable="YES"
howl_enable="YES"
EOF

service sniffle start
service snarl start
service howl start

Before we go on it’s a good moment to see if all the services are running:

# ps -aux | grep beam
howl    3135 571.1  1.3 1795680 108780  -  Ss   09:34    0:02.73 /usr/local/lib/howl/erts-8.3.5.1/bin/beam.smp -P 256000 -A 64 -W w -- -root /usr/local/lib/howl -progname howl -- -home /data/howl -- -boot /usr/local/lib/
snarl   3037 403.3  1.1 1803380  90676  -  Ss   09:33    0:03.15 /usr/local/lib/snarl/erts-8.3.5.1/bin/beam.smp -P 256000 -A 64 -W w -- -root /usr/local/lib/snarl -progname snarl -- -home /data/snarl -- -boot /usr/local/
sniffle 2487   4.1  1.3 1882760 112460  -  Is   09:33    0:04.03 /usr/local/lib/sniffle/erts-8.3.5.1/bin/beam.smp -P 256000 -A 64 -W w -- -root /usr/local/lib/sniffle -progname sniffle -- -home /data/sniffle -- -boot /us
root    1074   0.1  0.8 1734176  63308  -  Is   09:28    0:02.56 /usr/local/lib/chunter/erts-8.3.5.1/bin/beam.smp -P 256000 -A 64 -W w -- -root /usr/local/lib/chunter -progname chunter -- -home / -- -boot /usr/local/lib/
root     974   0.0  0.5 1691208  39304  -  Is   09:25    0:00.77 /usr/local/lib/fifo_zlogin/erts-8.3.5.1/bin/beam.smp -A30 -- -root /usr/local/lib/fifo_zlogin -progname usr/local/lib/fifo_zlogin/bin/fifo_zlogin -- -home

Configuring FiFo

So with everything set up we have to do two more configuration steps. First, we disable FiFo’s local dataset cache, which when configured uses LeoFS that isn’t set up yet. Second, we create an initial user and permission structure.

# disable leofs dataset cache
sniffle-admin config set storage.s3.host no_s3

# Initialize user
snarl-admin init default Project-FiFo Users admin <secret password>

# Add FreeBSD dataset repository
sniffle-admin datasets servers add https://bsd.project-fifo.net/images

Now we can go to our UI and take a look it should be at http://<your ip>.xip.io:8080, we are greeted by Cerberus’s login screen.

We need to configure a package. For that navigate to configure -> packages, let’s name it ‘small’, give it 100% CPU (this means 1 full core) a gigabyte of memory and 5 gigabytes of disk.

Next, we configure an IP range. Jails get a static IP address, IP Ranges can be understood as DHCP IP pools. We navigate to Configuration -> IP Ranges. Use the following settings that match the NAT’d network we setup earlier:

Name:       private
NIC Tag:    admin
VLAN:       0
Subnet IP:  192.168.1.0
Netmask:    255.255.255.0
Gateway:    192.168.1.1
First:      192.168.1.2
Last:       192.168.1.254

Note that we used ‘admin’ as nic tag, this relates to the setting in vmadm and does not need to be changed at this point.

Now we create a network, which is an abstraction over IP ranges to allow for fragmented or multi-range logical networks. It is somewhat straightforward it only has a name. Once created we edit the network to assign the IP Range we just created to it. Navigate to Configuration -> Network.

Last but not least we have to tell FiFo what datasets are available, for this go to the Datasets tab and import the FreeBSD 11.1 dataset.

The first jail

Now with everything set the only thing left to do is create our jail. Navigate to ‘Machines’ and fill out the fork with the dataset, package, and info we just inserted.

Once we click Create, we have to wait for a moment until the dataset is imported and the jail created. We can watch the state change through the states until it finally reaches running.

Now click on the little three dots on the right and select console. Your browser may complain about pop-ups if it does you have to allow them. This gives you a console to your newly created jail.