After failing over from the active cluster master to the redundant node (which holds the same configuration), 15 buckets report now
slave bucket=XXXXXX has unexpected mask=11 expected=0
This results in search factor not met for the corresponding indices.
I can see the cluster master running the "CMChangeMasksJob" in regular intervals but it looks to me it just can't handle those 15 buckets.
I am looking for any hints how to tackle this. First and foremost, what is a bucket mask? Are my buckets corrupted? Can I try to update the mask manually?
Simply resolved by restarting the Splunk instance running on the cluster master 🤣