We are seeing the same behavior in our 7 node search head cluster. All hardware is identical. Here are a few things that we've noticed. Nothing proven over time. Just observations at this point. running transfer shcluster-captain seems to kick the scheduler in a way where it temporarily redistributes scheduled searches to other nodes in the cluster. After a few minutes, the same node begins running all scheduled searches again. There is some ordering in play our search cluster nodes are named xyz001- xyz007, so the last digit is the only difference. a. if xyz006 is the captain, xyz007 ends up running all of the scheduled searches. b. if we transfer captaincy to xyz001, xyz002 runs the majority of the scheduled searches, then xyz003 picks up some of searches for about 5 to 10 minutes. Then xyz007 starts running all of the scheduled searches. We recently upgraded to a patched version of 6.3.3 (6.3.3.2) from 6.3.1, so these are our initial observations that may or may not be helpful. ... View more

splunk resync shcluster-replicated-config does a destructive resync, but in a way it does what you are looking for. If you edit savedsearches.conf on the captain and run splunk resync shcluster-replicated-config, the change will be replicated, but again, this is a destructive resync, so if a user makes a change via the UI on one of the nodes that is not currently the captain, the changes could be lost. We've used this method to migrate apps/users from standalone search heads to the search head cluster during maintenance periods. ... View more

I'm still in my initial testing, but gluster works if you mount the glusterfs via nfs locally. This isn't exactly ideal but seems to work better than straight NFS if you have good network connectivity and fast local storage. I've tested with both splunk 4.3 and 5.0.1. performance in 5.0.1 is comparable to standalone search heads. ... View more

Setting TRUNCATE = 999999 resolved the issue for me. Thanks again. ... View more

Thank you Gerald. I have SHOULD_LINEMERGE=false set in the props.conf. I will increase the the TRUNCATE option and report back with my results. We are running LWF, so the output buffer could be an issue, however, the data is not generated slowly, so that will be the next thing I look at. Rick ... View more