I've been fighting with Gluster with geo-replication on ZFS for several months now but I believe I've finally stumbled on the configuration that works best and gives much better Gluster performance on ZFS.
First a peek at the landscape
I probably don't have your typical storage needs. On this particular cluster I'm storing around 15 million files averaging 20MB each. No more than 255 files or sub-directories in a given directory.
My Gluster configuration uses a two-brick configuration with replica 2 (mirroring) and a single geo-replication slave over SSH.
Each Gluster brick uses ZFS RAID-Z volumes, spanning four SATA3 7200 RPM drives, and a single 256 GB SSD configured as a cache drive for the volume. Each Gluster server is connected on a private gigabit network segment to each other, and for the initial data load the Geo-replication server was connected to the same segment.
The Geo-replication slave also uses ZFS RAID-Z, but does not have a cache drive.
The goal with this configuration is to enable snapshot backups on either brick, plus geo-replication backups to a distant datacenter.