Backups With Project FiFo

I want to talk about one of the lesser known features of Project FiFo: Backups. FiFo has supported backups as far back as version 0.4.3. While not widely used, it is one of those features that can save your butt when things go seriously south.

FiFo’s Backups feature compliments the snapshot ability that has been in the system for quite some time now. While snapshots must remain on the hypervisor itself, backups are shipped to LeoFS storage. This has many benefits, it limits the number of snapshots, does not count against the local quota, and also protects against many different types of failures.

FiFo supports both full and incremental backups. Here are a few limitations and benefits to be aware of:

Limitations

• Backups that reside on the hypervisor are still snapshots and will count against a local quota.
• Once a backup is moved away from the hypervisor it can’t be restored without overwriting the current state.
• Restoring a backup might mean first deleting the local zfs volume for the vm.
• Incremental backups can only be made from a backup that is still on the hypervisor.

Benefits

• Ability to keep multiple branches with backups.
• Ability to make incremental backups – to choose between recovery speed vs space efficiency.
• Backups are stored as ZFS snapshots and additionally referenced within the zone.xml file allowing for manual restoration if things go seriously wrong.
• With backups, its possible to move back and forth in time – unlike snapshots, backups are not on the hypervisor. Even if the ZFS data on the hypervisor gets deleted, the backup still remains viable within LeoFS.

This all adds up to something quite awesome: allowing for proper grandfather backups concepts for VM’s! Backups can even be scripted using the fifo python client. Here is an example how this could be done.

Assume we want to use the following schedule:

• Monthly full backup.
• Weekly incremental backup
  • for the first week in a month use the monthly full backup as a basis
  • for other weeks use the prior week’s backup as a basis
• Daily incremental backup
  • for the first day of a week use that week’s backup as a basis
  • for the remaining days use the prior day’s backup as a basis

To allow for the incremental backups we need to keep some of the backups around:

• Monthly until the first weekly was done.
• Weekly until the next weekly was done but not longer the the next monthly.
• Daily until the next daily but not longer then the next weekly or monthly.

Fortunately the FiFo backup code helps a lot with this. An important part of the logic is ‘create a incremental snapshot and delete its parent from the hypervisor’ and that is exactly the behavior of the backup code when passing both a parent and requesting a delete.

Code for this can be found here.

But that is not all.

Since backups are not being stored on the hypervisor, we can now recover from critical hardware failures such as faulty disk arrays, exploding motherboards etc. With some manual configuration and work setting up LeoFS for datacenter to datacenter replication – even a datacenter failure can be recovered from.