Best-Effort Extent-Same, a btrfs dedupe agent
GitHub user Zygo has released bees, a block-oriented userspace deduplication agent designed for btrfs filesystems, offering space-efficient hash tables and daemon mode operation.
bees is designed to recover free space in btrfs filesystems through whole-filesystem deduplication. The tool scans data only once, even with snapshots and reflinks, and features a space-efficient hash table that can use as little as 1 GB per 10 TB of unique data. It operates in daemon mode, incrementally deduplicating new data as it appears without requiring scheduled downtime.
Key Features
The tool prioritizes largest extents first to recover more free space during maintenance windows and works with btrfs compression. It maintains a constant hash table size regardless of dataset growth and includes automatic self-throttling to reduce system load. bees also supports persistent hash tables for rapid restarts after shutdown.
Limitations
bees has no include/exclude filters and doesn't accept file lists. It requires root privileges (CAP_SYS_ADMIN plus filesystem read/modify capabilities) and is btrfs-only. The first run may increase metadata space usage if many snapshots exist, and the hash table size doesn't decrease if the dataset shrinks.
Usage
The tool is available under GPL version 3 or later and can be installed from GitHub. Users should be aware of potential btrfs kernel bugs, especially DATA CORRUPTION WARNING for old kernels. Bug reports and patches can be sent to Zygo Blaxell at [email protected] or submitted via GitHub.
Related
