Alerting

How can I set up an alert for Splunk errors?

Explorer

Splunk only notifies of errors like file system permission issues in the top right messages dropdown.

Since I rarely use Splunk web interface, I'm always missing them.

Is there a way to get notified of these errors? Can I set up alerts for them?

0 Karma
1 Solution

SplunkTrust
SplunkTrust

I've built a library of alerts into an app called "Alerts For Splunk Admins (on SplunkBase)" or github link , various alerts address a large number of potential "ERROR" scenarios in Splunk.

The app was mostly built by looking for ERROR/WARNING in the logs and identifying which ones caused a system or user impact of some kind, however if you want to look for just error/warning in the internal logs you likely want to run something like:

index=_internal sourcetype=splunkd source=*splunkd.log ERROR OR WARN | cluster show_count=true

And then you get a giant list of entries that mention error or warning, you may also want to watch the scheduler.log and the mongod.log, however in mongodb you look for " E " OR " F "

View solution in original post

SplunkTrust
SplunkTrust

I've built a library of alerts into an app called "Alerts For Splunk Admins (on SplunkBase)" or github link , various alerts address a large number of potential "ERROR" scenarios in Splunk.

The app was mostly built by looking for ERROR/WARNING in the logs and identifying which ones caused a system or user impact of some kind, however if you want to look for just error/warning in the internal logs you likely want to run something like:

index=_internal sourcetype=splunkd source=*splunkd.log ERROR OR WARN | cluster show_count=true

And then you get a giant list of entries that mention error or warning, you may also want to watch the scheduler.log and the mongod.log, however in mongodb you look for " E " OR " F "

View solution in original post

Explorer

Thanks, this looks great! Would expect Splunk to do such a basically needed feature themselves.

0 Karma

SplunkTrust
SplunkTrust

Ideally I'd like to see the majority of the alerts I created built into Splunk in some form (preferably with a 'notify me by email button')!

To be fair, some of them such as when you have buckets rolling too fast or similar are now shown in the console in 7.2 which is great, previously you had to be watching the logs to know about this...however I'd like to see a lot more built into Splunk

Anyway hopefully it helps, good luck!

0 Karma

Communicator

You can look for internal errors with something like

index=_internal log_level=ERROR

And produce Alerts for the results.

0 Karma

SplunkTrust
SplunkTrust

The banner messages are available in logs and can be seen using

|rest /services/messages

You can create an alert based on what you need from the above to alert for new messages.

0 Karma