Can I be notified or alerted when Splunk goes down?

[Dark_Ichigo]

I want to be alerted when Splunkd goes down, how can I be notified?

[ChrisG]

See the previous answer about setting up an email alert if Splunk is down in production.

[Dark_Ichigo]

I'm going to try creating a cronjob that can monitor Splunks status, but as from what I can see, there doesn't seem to be a perfect (risk free) way of doing this.

All other applications I have had to deploy in my lifetime had alarming capabilities for this purpose, as i kinda makes sense to have it.

[Drainy]

Well played sir, well played

[martin_mueller]

Splunk for everything 😛

[Drainy]

But why use Splunk? You'd be better using a dedicated monitoring solution or some other cronjob to monitor for it

[martin_mueller]

Without a second system watching the first you won't notice the first system disappearing entirely, for instance when the machine goes boom or when someone takes Splunk down by using a sledgehammer...

[Drainy]

Why use another instance? Why not just do it the usual way of a cronjob checking every X minutes and firing an email if its down/trying to restart it?

[martin_mueller]

If they both go down simultaneously you likely have a larger problem.

[Dark_Ichigo]

What if they both go down?

[martin_mueller]

Just run a second Splunk on a tiny system somewhere (VM?) and have each monitor each other, raising hell when one goes missing.

[Dark_Ichigo]

Yes I understand that, but there should be an alarm that's raised and is able to notify Operations when its down...

[martin_mueller]

When Splunk is down there is nothing Splunk can do because, well, it's down.

[Dark_Ichigo]

Yes but that means there isn't anything application specific ready to provide a user with alerts or notifications if and when Splunk does go down.