topic Re: How can I set up an alert for Splunk errors? in Alerting

How can I set up an alert for Splunk errors?

reallyliri — Mon, 25 Feb 2019 09:29:16 GMT

Splunk only notifies of errors like file system permission issues in the top right messages dropdown.

Since I rarely use Splunk web interface, I'm always missing them.

Is there a way to get notified of these errors? Can I set up alerts for them?

Re: How can I set up an alert for Splunk errors?

lakshman239 — Mon, 25 Feb 2019 14:23:49 GMT

The banner messages are available in logs and can be seen using

|rest /services/messages

You can create an alert based on what you need from the above to alert for new messages.

Re: How can I set up an alert for Splunk errors?

guarisma — Mon, 25 Feb 2019 16:55:27 GMT

You can look for internal errors with something like

index=_internal log_level=ERROR

And produce Alerts for the results.

Re: How can I set up an alert for Splunk errors?

gjanders — Mon, 25 Feb 2019 23:23:22 GMT

I've built a library of alerts into an app called "Alerts For Splunk Admins (on SplunkBase)" or github link , various alerts address a large number of potential "ERROR" scenarios in Splunk.

The app was mostly built by looking for ERROR/WARNING in the logs and identifying which ones caused a system or user impact of some kind, however if you want to look for just error/warning in the internal logs you likely want to run something like:

index=_internal sourcetype=splunkd source=*splunkd.log ERROR OR WARN | cluster show_count=true

And then you get a giant list of entries that mention error or warning, you may also want to watch the scheduler.log and the mongod.log, however in mongodb you look for " E " OR " F "

Re: How can I set up an alert for Splunk errors?

reallyliri — Tue, 26 Feb 2019 06:20:35 GMT

Thanks, this looks great! Would expect Splunk to do such a basically needed feature themselves.

Re: How can I set up an alert for Splunk errors?

gjanders — Tue, 26 Feb 2019 08:14:42 GMT

Ideally I'd like to see the majority of the alerts I created built into Splunk in some form (preferably with a 'notify me by email button')!

To be fair, some of them such as when you have buckets rolling too fast or similar are now shown in the console in 7.2 which is great, previously you had to be watching the logs to know about this...however I'd like to see a lot more built into Splunk

Anyway hopefully it helps, good luck!