I am working on "DMC Alert - Search Peer Not Responding" on master node. The query that it uses is as follows -
| rest splunk_server=local /services/search/distributed/peers/
| where status!="Up"
| fields peerName, host, status
| rename peerName as Instance, status as Status
The issue is - once the peer goes down, that peer gets removed from the table where this query does its search. You can see this happening in "Settings --> Distributed Search --> Search Peers". So even though the peer is down, the query does not return any result and thus does not generate an alert. Is there any fix to this?
I found another table from which the peer does not get removed when it goes down. "Settings --> Distributed Management Console --> Instances (tab next to Overview)". Can anyone suggest what query should I run so that it searches for the peer status in this table?
... View more