You encountered a situation that, while confusing, is
a common and painful mistake when using nullfs mounts.
You had previously configured your system to mount
your actual /home directory into /compat/ubuntu/home
using the nullfs filesystem. This created a mirrored
view of your home directory inside the Linux compatibility
environment.
When you later issued the command sudo rm -rf /compat,
you unintentionally deleted everything inside that path.
However, because /compat/ubuntu/home was just another
view of your real /home, the command also erased the
contents of your home directory. The removal was not
isolated to the compatibility layer; it affected your
actual data because the two paths referenced the same
underlying files. This behavior is expected with nullfs,
but not always obvious.
You asked why the ZFS Boot Environment did not restore
the deleted files. The answer lies in how Boot
Environments function. A Boot Environment is essentially
a snapshot of the ZFS root dataset, typically
zroot/ROOT/default. This snapshot captures the state of
files that reside within that dataset.
Your /home directory, however, is very likely located
on a separate dataset, such as zroot/home, which is not
automatically included in the snapshot of the root
dataset. Therefore, when you rolled back the Boot
Environment, the root filesystem reverted to an earlier
state, but the separate home dataset remained in its
current, post-deletion state.
This explains why your system appeared restored, yet
your files were still missing.
If no snapshot of the zroot/home dataset exists from
before the deletion, then unfortunately your home
directory data cannot be restored using ZFS.
You may verify whether such a snapshot exists by running
zfs list -t snapshot | grep home. If any relevant
snapshots are listed, they may be rolled back or cloned
for recovery. If none exist, your options become severely
limited.
There is no official undelete tool for ZFS that can
recover deleted data without a snapshot. The ZFS design
prioritizes data integrity and consistency over
accidental recovery.
Advanced recovery using tools such as zdb -X is
possible, but extremely difficult and often impractical
for typical users. File recovery tools like photorec
or testdisk do not understand ZFS metadata and are
unlikely to help in your situation.
In conclusion, your data was deleted because a
nullfs-mounted path pointed directly to it, and the
system followed the deletion command faithfully.
The Boot Environment rollback did not help because it did
not cover the dataset that held your home directory.
If you have no snapshot of that dataset, then recovery
may not be feasible without specialized ZFS forensic work.
