How do you report on the search head cluster availability score (99.9999% score)?

[gduggan1]

I am tasked with reporting on our Splunk environment. I am running a Search Head cluster with 3 Search Heads and an index cluster with 6 indexers (single site). They are all on 6.6.3.

Does anyone know a search that would return a result indicating the availability of the SH cluster and the Index cluster independent of each other. I am looking for a (99.99999%) 5x9's availability score specifically. Up until now I am just showing up times on my search heads (based on last restart of splunkd).

I wasn't able to find this in the monitoring console either (DMC). Any help would be greatly appreciated 🙂

[lycollicott]

Oh, the magic nines.

It is a little suspect whenever a monitoring system has to monitor itself. For example, all of your search heads could be up, but your network could be down.

But you could try this app https://splunkbase.splunk.com/app/1493/ on your DMC to monitor the search heads. If you leave splunkweb enabled on your indexers (which most people turn off) you could also use it for those.....just keep the port & url known only to yourself and fellow admins.

Caveat: I have not used that app.

[gduggan1]

Awesome! thanks for the app i'll test it out, we are currently running a Round Robin DNS on the front end but will be moving to an F5 soon. I could prob integrate the 2 measures to have a somewhat specific measure of availability. I guess this just scratches the surface though because we have to take in account "can you search the data you are looking for" measures as well....

[lycollicott]

The F5 might have some reporting functions that could be useful, too.

As for searchable.....theoretically you could schedule a shell script that would do a command line search every few minutes and alert when results are 0 or an error.