I am tasked with reporting on our Splunk environment. I am running a Search Head cluster with 3 Search Heads and an index cluster with 6 indexers (single site). They are all on 6.6.3.
Does anyone know a search that would return a result indicating the availability of the SH cluster and the Index cluster independent of each other. I am looking for a (99.99999%) 5x9's availability score specifically. Up until now I am just showing up times on my search heads (based on last restart of splunkd).
I wasn't able to find this in the monitoring console either (DMC). Any help would be greatly appreciated 🙂
Oh, the magic nines.
It is a little suspect whenever a monitoring system has to monitor itself. For example, all of your search heads could be up, but your network could be down.
But you could try this app https://splunkbase.splunk.com/app/1493/ on your DMC to monitor the search heads. If you leave splunkweb enabled on your indexers (which most people turn off) you could also use it for those.....just keep the port & url known only to yourself and fellow admins.
Caveat: I have not used that app.
Awesome! thanks for the app i'll test it out, we are currently running a Round Robin DNS on the front end but will be moving to an F5 soon. I could prob integrate the 2 measures to have a somewhat specific measure of availability. I guess this just scratches the surface though because we have to take in account "can you search the data you are looking for" measures as well....
The F5 might have some reporting functions that could be useful, too.
As for searchable.....theoretically you could schedule a shell script that would do a command line search every few minutes and alert when results are 0 or an error.