This impacts search head snapshots which fail with too long path names. Then if a shcluster member falls out of sync far enough where only a resync can recover, old snapshots will be invoked wiping out changes to the cluster since the last snapshot. Don't resync to recover an unhealthy member until you check /var/run/snapshot and ensure you have current bundles on at least the captain.
There's close to zero documentation about snapshots and how they work other than a little note in release notes and snapshots failing due to long pathnames. We of course will add to look for the errors regarding long pathnames and snapshots failing in splunkd.log in our monitoring solution, but pop up errors/messages in the gui, like we get for other cluster problems would have really helped here.
... View more