ZREPL is a great tool for creating snapshots of your ZFS filesystems and then copying them to a remote ZFS storage server. I have been using ZREPL for some time and I recently relied upon it to restore filesystems. However, the experience revealed some gotchas.

The first time I had to rollback a filesystem from a local snapshot I found ZREPL didn’t want to continue replicating that host afterwards. I got into a mess with ZREPL’s ZFS holds and bookmarks and I ended up deleting all the snapshots on both sides of the replication, losing historical data that I wanted to keep.

A few weeks later I had to use a remote ZREPL snapshot to restore a laptop that needed a new hard drive. The restoration was successful, but getting ZREPL to continue replicating this machine was not. I manually trawled through the ZFS holds and bookmarks, clearing them as needed. I lost a few historical snapshots that I wanted to keep in the process and thought that there must be a better way. Using ZREPL after recoveries was proving to be less than ideal. I have found that better way and it is so simple that you will remember it.

Yesterday, I had to do another recovery. This time on a experimental server that I really should never have built using old SATA hard drives. The zpool had three drives in the pool, two mirrored and an online spare. When one of the mirrored drives failed, ZFS successfully resilvered to the spare. Before I had an opportunity to replace the failed drive, the other mirrored drive failed. It ran OK for a while on parity, but when I shut the machine down to replace the original failed drive, the server would not boot.

As I had hourly ZREPL replicas stored remotely and three more old replacement drives available, I was confident that this problem would not be a disaster. I replaced all three drives, reinstalled the OS and used ZFS send/recv to restore the filesystems from my remote ZREPL replica. The experimental server was operational, but ZREPL was no longer creating remote replicas as it should.

The following describes a simple method that I now use to do a full zpool restore. It is equally relevant if you have to restore an individual filesystem that has been replicated remotely by ZREPL either by rolling back a local snapshot or sending from a remote replica.

How To Restore Using ZREPL

Example Scenario

jewel.example.com
This is the host that needs a filesystem restoration. Hostname abbreviated to ‘jewel’.

zroot
This is the ZFS pool on ‘jewel’ that has a Root on ZFS filesystem containing the important data that must be restored.

safe.example.com
This is the ZREPL storage server. Hostname abbreviated to ‘safe’.

zdata
This is the ZFS pool on ‘safe’ that contains the ZFS snapshots that were replicated remotely by ZREPL.

Tip 1:

Always have a bootable recovery image available. It can be a USB datastick, a live CD or PXE boot disk. It should have the same OS (and version) as the system to be recovered. It must contain all of the necessary ZFS components so that you can import your ZFS pool without mounting the file systems. This enables you to restore a ZFS on Root system. Remember to update this recovery boot image whenever you upgrade the OS or ZFS features.

Tip 2:

Use a script that is run from Cron or Periodic to regularly create a text file report about your ZFS filesystems and their mountpoints. If you have access to a private git repository, have the script ‘add, commit and push’ the report to the repository. If you don’t have a private git repo, have the machine email you the reports instead. You will be thankful that you have the mountpoint information when you most need it!

After a fresh operating system install and the recreation of the zpool(s) required (You had that info kept safe in your private git repo), boot the host using the recovery disk image.

Establish that you can ssh as root between the remote ZREPL storage and the host to be recovered using passwordless certificate login. Get this proven to run in both directions first before doing anything else. If you are preparing your recovery image, consider using a hardware token authenticator like a Yubikey to store the private keys which is safer than leaving private keys for root on a bootable USB stick!

Login as root on jewel.example.com to import the empty ZFS pool that you have recently created as part of the OS install. The -f option is required to import a pool that has been previously mounted by another system.

root@jewel$ zpool import -f zroot

Login as root on safe.example.com to identify the snapshot(s) that you wish to restore

root@safe$ zfs list -t snapshot -H -o name \
| grep jewel.example.com \
| grep 20260212_11

-t snapshot : Lists only snapshots
-H : Strips the headers from the output
-o name : Includes only the snapshot names in the output
grep jewel.example.com : Reduces the output to reference only the machine of interest
grep 20260212_11 : Reduces the output to snapshots taken on 12 February 2026 in the hour of 11am

Select the latest snapshot that you know will contain everything that you want to restore. This will be one that completed successfully before the disaster.  Use ZFS Send/Recv over SSH to restore the filesystem.

root@safe$ zfs send -r zdata/zrepl/jewel.example.com/zroot@zrepl_20260212_1132_000 \
| pv \
| ssh root@jewel.example.com "zfs recv -Fu zroot"

Piping through ‘pv’ is optional. It shows progress of the data transfer, otherwise you just have to wait for the prompt to return on your terminal to know that it has finished.

If you cannot or do not want to perform a recursive restore as shown above, omit the -r option and send filesystems individually.

When the filesystem restoration has completed, shutdown the restored host and remove the USB live image recovery datastick.

root@jewel$ poweroff

Boot the restored host and check that all is OK and as expected.

Temporarily stop the ZREPL service at the end that drives the replica transfer. If ‘jewel’ is configured to push replicas to ‘safe’, stop ZREPL on jewel. Otherwise, stop ZREPL on safe.

The simple action that prevents ZREPL problems

Login as root on safe.example.com, now rename the dataset holding the replicas for jewel
root@safe$ zfs rename zdata/zrepl/jewel.example.com/zroot zdata/zrepl/jewel-20260212/zroot
All of that dataset’s snapshots will automatically be renamed to correspond with the new name of their parent.

Restart ZREPL and manually wake up a replication. The process will automatically create a new dataset called zdata/zrepl/jewel.example.com/zroot . This will allow ZREPL to continue to operate as normal, effectively starting afresh.

All of your historical snapshots for jewel will now be in zdata/zrepl/jewel-20260212/zroot and its descendants. These will not be pruned automatically. You can still use them for restores, but they are now outside of ZREPL.

If you want to recover disk space, delete the snapshots that you no longer need from zdata/zrepl/jewel-20260212/zroot.

Re-enable and restart ZREPL and you should find that replication is back to normal.