While researching my zfs problem, I stumbled over this: http://martinliu.cn/workshop-redhat-storage-splunk/
It seems that this does the trick:
echo OPTIMISTIC_ABOUT_FILE_LOCKING = 1 >> /opt/splunk/etc/splunk-launch.conf
I'm still in my initial testing, but gluster works if you mount the glusterfs via nfs locally. This isn't exactly ideal but seems to work better than straight NFS if you have good network connectivity and fast local storage. I've tested with both splunk 4.3 and 5.0.1. performance in 5.0.1 is comparable to standalone search heads.
I couldn't get search head pooling to work with gluster 3.2.5 😞
Error in search head pooling enable: Failed to lock /mnt/splunk/etc/etc/users/testpath with return code -1: File exists
I logged a call with Splunk support and they confirmed that it is not supported... feature request submitted.
I'm going to use DRBD in the meantime, but gluster would have been way better!