Monitoring Cluster health with a search?

[LiquidTension]

Currently I have a clustered environment with 4 indexers. I have 2 two cases run into scenarios where the cluster master indicates that a host is "down." When I go to the indexer in question though, splunkd is running so targeting that with a tool to monitor processes (via SCOM) is useless.

My question is what are people doing to monitor for these typses of circumstances?

[LiquidTension · Accepted solution]

index="_internal" host=mon-* Splunkd daemon is not responding

This search helped in for scenario's like the one I mentioned above.