We have the following alert -
| rest splunk_server=local /services/search/distributed/peers/
| where status!="Up"
| fields peerName, status
| rename peerName as Instance, status as Status
Recently, the indexer cluster master has been marked as down consistently even though is has been up.
What can it be?
Make sure that the ports listed in the Splunk Common Network Ports diagram are open in the way that they should be according to your cluster topology.
http://downloads.jordan2000.com/splunk/Splunk-Common-Network-Ports-v2.0.3.png
The issue is intermittent and occasionally we see this alert... and it's consistent in the sense that every other day or so we get this alert.
The ports have been open for a long while.
What does your topology look like for the cluster?