topic Re: Can I be notified or alerted when Splunk goes down? in Getting Data In

Can I be notified or alerted when Splunk goes down?

Dark_Ichigo — Mon, 25 Feb 2013 23:28:24 GMT

I want to be alerted when Splunkd goes down, how can I be notified?

Re: Can I be notified or alerted when Splunk goes down?

ChrisG — Mon, 25 Feb 2013 23:55:07 GMT

See the previous answer about setting up an email alert if Splunk is down in production.

Re: Can I be notified or alerted when Splunk goes down?

Dark_Ichigo — Tue, 26 Feb 2013 01:59:08 GMT

Yes but that means there isn't anything application specific ready to provide a user with alerts or notifications if and when Splunk does go down.

Re: Can I be notified or alerted when Splunk goes down?

martin_mueller — Tue, 26 Feb 2013 09:11:01 GMT

When Splunk is down there is nothing Splunk can do because, well, it's down.

Re: Can I be notified or alerted when Splunk goes down?

Dark_Ichigo — Tue, 26 Feb 2013 22:42:37 GMT

Yes I understand that, but there should be an alarm that's raised and is able to notify Operations when its down...

Re: Can I be notified or alerted when Splunk goes down?

martin_mueller — Wed, 27 Feb 2013 07:32:00 GMT

Just run a second Splunk on a tiny system somewhere (VM?) and have each monitor each other, raising hell when one goes missing.

Re: Can I be notified or alerted when Splunk goes down?

Dark_Ichigo — Wed, 27 Feb 2013 07:44:30 GMT

What if they both go down?

Re: Can I be notified or alerted when Splunk goes down?

martin_mueller — Wed, 27 Feb 2013 08:11:50 GMT

If they both go down simultaneously you likely have a larger problem.

Re: Can I be notified or alerted when Splunk goes down?

Drainy — Wed, 27 Feb 2013 08:41:42 GMT

Why use another instance? Why not just do it the usual way of a cronjob checking every X minutes and firing an email if its down/trying to restart it?

Re: Can I be notified or alerted when Splunk goes down?

martin_mueller — Wed, 27 Feb 2013 09:12:05 GMT

Without a second system watching the first you won't notice the first system disappearing entirely, for instance when the machine goes boom or when someone takes Splunk down by using a sledgehammer...

Re: Can I be notified or alerted when Splunk goes down?

Drainy — Wed, 27 Feb 2013 10:02:34 GMT

But why use Splunk? You'd be better using a dedicated monitoring solution or some other cronjob to monitor for it

Re: Can I be notified or alerted when Splunk goes down?

martin_mueller — Wed, 27 Feb 2013 10:05:12 GMT

Splunk for everything 😛

Re: Can I be notified or alerted when Splunk goes down?

Drainy — Wed, 27 Feb 2013 10:29:00 GMT

Well played sir, well played

Re: Can I be notified or alerted when Splunk goes down?

Dark_Ichigo — Wed, 27 Feb 2013 23:23:01 GMT

I'm going to try creating a cronjob that can monitor Splunks status, but as from what I can see, there doesn't seem to be a perfect (risk free) way of doing this.

All other applications I have had to deploy in my lifetime had alarming capabilities for this purpose, as i kinda makes sense to have it.