How can I set up an alert for Splunk errors?

[reallyliri]

Splunk only notifies of errors like file system permission issues in the top right messages dropdown.

Since I rarely use Splunk web interface, I'm always missing them.

Is there a way to get notified of these errors? Can I set up alerts for them?

[gjanders · Accepted solution]

I've built a library of alerts into an app called "Alerts For Splunk Admins (on SplunkBase)" or github link , various alerts address a large number of potential "ERROR" scenarios in Splunk.

The app was mostly built by looking for ERROR/WARNING in the logs and identifying which ones caused a system or user impact of some kind, however if you want to look for just error/warning in the internal logs you likely want to run something like:

index=_internal sourcetype=splunkd source=*splunkd.log ERROR OR WARN | cluster show_count=true

And then you get a giant list of entries that mention error or warning, you may also want to watch the scheduler.log and the mongod.log, however in mongodb you look for " E " OR " F "

[reallyliri]

Thanks, this looks great! Would expect Splunk to do such a basically needed feature themselves.

[gjanders]

Ideally I'd like to see the majority of the alerts I created built into Splunk in some form (preferably with a 'notify me by email button')!

To be fair, some of them such as when you have buckets rolling too fast or similar are now shown in the console in 7.2 which is great, previously you had to be watching the logs to know about this...however I'd like to see a lot more built into Splunk

Anyway hopefully it helps, good luck!

[guarisma]

You can look for internal errors with something like

index=_internal log_level=ERROR

And produce Alerts for the results.

[lakshman239]

The banner messages are available in logs and can be seen using

|rest /services/messages

You can create an alert based on what you need from the above to alert for new messages.