Is it normal behavior for the Cluster Master to transition from=Searchable to=SearchablePendingMask?

[omgwut56k]

We have a 54 node cluster and we are seeing these events streaming continuously every few minutes from each node. Can anyone comment if this is normal activity for the CM, or an issue? Can anyone tell me what we are seeing here?

Thank you

09-15-2015 10:22:52.306 -0400 INFO  CMPeer - peer=140E75CA-9E39-4992-B857-C34200944483 peer_name=server bid=index_name~3494~8DA53A52-7E20-4178-BB4D-9210C489FBCA transitioning from=Searchable to=SearchablePendingMask oldmask=0x0 newmask=0x4 reason="swap primaries"

[dxu_splunk · Accepted solution]

these shouldn't be happening continuously (it can happen for a while as the cluster fixes itself up but should stop once that finishes).

are the peers readding themselves into the cluster often? (search for event=addPeer).