This tutorial covers a simple setup of FiFo inside of Digital Oceans, starting with the installation continuing with configuration and finally creating our first jail.
Disclaimer
Before you get too excited, I want to put the bad news in the beginning DO’s private networks do not allow for multicast traffic which means FiFo’s node discovery does not work beyond a single node by default. There are a few ways around that (like tunnels or VPN’s), but that’d be beyond the scope of this little write-up.
Now the good news, FiFo works flawlessly on Digital Oceans FreeBSD image, the installation takes just a few minutes, and it works stable. The whole process takes a bit less than 30 minutes, most of that time spent compiling the kernel to enable RCTL and VNET which does not come enabled by default.
The tutorial is not meant to create a production system, security concerns are woefully ignored. This is to spend a few hours playing with FiFo and then tearing it down again. If you plan to run FiFo in production please follow the official documentation or contact us for commercial support.
Creating the Droplet
For this example, we chose the FreeBSD 11.1 ZFS image and the 8GB / 4 CPU droplet so compiling doesn’t take too much time.
Select your location (I like Frankfurt it’s nice and close), and the SSH key to use. Because this will be a single node system there is no need for the “Private networking” option. With all the desired settings selected go ahead and create your droplet. It’ll take a minute or two to boot.
Configuring the system
Now we just install FiFo as covered in the documentation.
We download the kernel code. The command is:
fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/`freebsd-version -k`/src.txz -o /tmp/src.txz tar -C / -xzf /tmp/src.txz
This will take a moment or two, there are lots of files to download. Once we’re done, we’ll make our own kernel with VNET and RCTL, this too is described in the documentation, but for brevity here we go. We start with creating a custom configuration.
cd /usr/src/sys/amd64/conf cat > FIFOKERNEL <<EOL include GENERIC ident FIFOKERNEL options VIMAGE # VNET/Vimage support options RACCT # Resource containers options RCTL # same as above EOL cd /usr/src
After we’ve set up the configuration, we build and then install the kernel. Again, this commands will take a few minutes so grab a coffee while you wait.
# Build the kernel make -j4 buildkernel KERNCONF=FIFOKERNEL # Install the new kernel make -j4 installkernel KERNCONF=FIFOKERNEL
With the kernel installed, we set up a few modules to be loaded on the next boot. Namely the Linux emulator the and related modules as well as well as enabling RCTL. After that, we reboot our droplet.
cat <<EOF >> /boot/loader.conf linux64_load="YES" linux_load="YES" fdescfs_load="YES" linprocfs_load="YES" linsysfs_load="YES" tmpfs_load="YES" mac_portacl_load="YES" kern.racct.enable=1 EOF reboot
Once the system is back up, we can double check that our kernel was loaded.
uname -a FreeBSD fifo-bsd 11.1-RELEASE-p1 FreeBSD 11.1-RELEASE-p1 #0 r324521: Wed Oct 11 09:12:27 UTC 2017 root@fifo-bsd:/usr/obj/usr/src/sys/FIFOKERNEL amd64
All good, next up we create a few ZFS datasets to organize jails and FiFo data.
zfs create zroot/data zfs set mountpoint=/data zroot/data zfs create zroot/data/sniffle zfs create zroot/data/snarl zfs create zroot/data/howl zfs create zroot/jails zfs set mountpoint=/zroot/jails zroot/jails
* Now is a good time to make a snapshot using the DO dashboard if you plan on going through the setup more than once.
Configuring Networking
We need to configure some networking. FiFo uses a bridge to attach VNETS, we can create it using:
cat <<EOF >> /etc/rc.conf cloned_interfaces="bridge0" autobridge_interfaces="bridge0" autobridge_bridge0="vtnet0" ifconfig_bridge0="inet 192.168.1.1/24" EOF
For Jails to have real internet access you’ll need a NAT set up, we will make a simple NAT with PF but this should only be used for testing. Also of course this only allows traffic out, in a production setup an ip pool for jails is best practice.
cat <<EOF > /etc/pf.conf
IP_PUB="`ifconfig vtnet0 | grep "inet " | grep -v "inet 10" | cut -d: -f2 | cut -d\ -f2`"
NET_JAIL="192.168.1.0/24"
scrub in all
nat pass on vtnet0 from \$NET_JAIL to any -> \$IP_PUB
pass out log on vtnet0 proto { tcp, udp, icmp } all
EOF
cat <<EOF >> /etc/rc.conf
pf_enable="YES"
gateway_enable="YES"
EOF
/etc/netstart
pfctl -nf /etc/pf.conf
service pf start
Installing the base components
Project-FiFo provides a pkg repository so installing the components is quite easy. To get the packages, we first tell the system about our new repository.
mkdir -p /usr/local/etc/pkg/repos
cat <<EOF > /usr/local/etc/pkg/repos/ProjectFiFo.conf
ProjectFiFo: {
url: "pkg+https://freebsd.project-fifo.net/rel/amd64/11.0",
mirror_type: "srv",
enabled: yes
}
EOF
pkg update
Now we install the base services, those are required on every hypervisor not only managing systems. Since we run all on one host, it doesn’t make a huge difference, but let’s do it correctly.
pkg install vmadm chunter zlogin cat <<EOF >> /etc/rc.conf zlogin_enable="YES" chunter_enable="YES" vmadm_enable="YES" EOF
Since the DO FreeBSD comes with an unusual configuration of two IP’s on the same interface, FiFo can’t automatically detect the network configuration. So before we start the services we need to make sure chunter’s config file exists. Since we want everything on one host it’s rather easy we can just copy over the example.
cp /usr/local/lib/chunter/etc/chunter.conf.example /usr/local/lib/chunter/etc/chunter.conf
Now we start the services.
service zlogin start service chunter start
Installing the management system
After the base services are installed and started we install the management system, this too is done via packages.
pkg install bash fifo-sniffle fifo-snarl fifo-howl fifo-cerberus
Before we start the services, we configure howl to run on 8080 and 8443, so we do not require privileged ports. We do this by editing /data/howl/etc/howl.conf and changing the following lines:
## The port howl listens on for websockets ## ## Default: 80 ## ## Acceptable values: ## - an integer http_port = 8080 ## The port howl listens to. ## ## Default: 443 ## ## Acceptable values: ## - an integer ssl.port = 8443
Now we can boot up the system, the start order doesn’t matter as the components are self-organizing.
cat <<EOF >> /etc/rc.conf sniffle_enable="YES" snarl_enable="YES" howl_enable="YES" EOF service sniffle start service snarl start service howl start
Before we go on it’s a good moment to see if all the services are running:
# ps -aux | grep beam howl 3135 571.1 1.3 1795680 108780 - Ss 09:34 0:02.73 /usr/local/lib/howl/erts-8.3.5.1/bin/beam.smp -P 256000 -A 64 -W w -- -root /usr/local/lib/howl -progname howl -- -home /data/howl -- -boot /usr/local/lib/ snarl 3037 403.3 1.1 1803380 90676 - Ss 09:33 0:03.15 /usr/local/lib/snarl/erts-8.3.5.1/bin/beam.smp -P 256000 -A 64 -W w -- -root /usr/local/lib/snarl -progname snarl -- -home /data/snarl -- -boot /usr/local/ sniffle 2487 4.1 1.3 1882760 112460 - Is 09:33 0:04.03 /usr/local/lib/sniffle/erts-8.3.5.1/bin/beam.smp -P 256000 -A 64 -W w -- -root /usr/local/lib/sniffle -progname sniffle -- -home /data/sniffle -- -boot /us root 1074 0.1 0.8 1734176 63308 - Is 09:28 0:02.56 /usr/local/lib/chunter/erts-8.3.5.1/bin/beam.smp -P 256000 -A 64 -W w -- -root /usr/local/lib/chunter -progname chunter -- -home / -- -boot /usr/local/lib/ root 974 0.0 0.5 1691208 39304 - Is 09:25 0:00.77 /usr/local/lib/fifo_zlogin/erts-8.3.5.1/bin/beam.smp -A30 -- -root /usr/local/lib/fifo_zlogin -progname usr/local/lib/fifo_zlogin/bin/fifo_zlogin -- -home
Configuring FiFo
So with everything set up we have to do two more configuration steps. First, we disable FiFo’s local dataset cache, which when configured uses LeoFS that isn’t set up yet. Second, we create an initial user and permission structure.
# disable leofs dataset cache sniffle-admin config set storage.s3.host no_s3 # Initialize user snarl-admin init default Project-FiFo Users admin <secret password> # Add FreeBSD dataset repository sniffle-admin datasets servers add https://bsd.project-fifo.net/images
Now we can go to our UI and take a look it should be at http://<your ip>.xip.io:8080, we are greeted by Cerberus’s login screen.
We need to configure a package. For that navigate to configure -> packages, let’s name it ‘small’, give it 100% CPU (this means 1 full core) a gigabyte of memory and 5 gigabytes of disk.
Next, we configure an IP range. Jails get a static IP address, IP Ranges can be understood as DHCP IP pools. We navigate to Configuration -> IP Ranges. Use the following settings that match the NAT’d network we setup earlier:
Name: private NIC Tag: admin VLAN: 0 Subnet IP: 192.168.1.0 Netmask: 255.255.255.0 Gateway: 192.168.1.1 First: 192.168.1.2 Last: 192.168.1.254
Note that we used ‘admin’ as nic tag, this relates to the setting in vmadm and does not need to be changed at this point.
Now we create a network, which is an abstraction over IP ranges to allow for fragmented or multi-range logical networks. It is somewhat straightforward it only has a name. Once created we edit the network to assign the IP Range we just created to it. Navigate to Configuration -> Network.
Last but not least we have to tell FiFo what datasets are available, for this go to the Datasets tab and import the FreeBSD 11.1 dataset.
The first jail
Now with everything set the only thing left to do is create our jail. Navigate to ‘Machines’ and fill out the fork with the dataset, package, and info we just inserted.
Once we click Create, we have to wait for a moment until the dataset is imported and the jail created. We can watch the state change through the states until it finally reaches running.
Now click on the little three dots on the right and select console. Your browser may complain about pop-ups if it does you have to allow them. This gives you a console to your newly created jail.
Leave a Reply