Alerting

How can I set up an alert for Splunk errors?

reallyliri
Explorer

Splunk only notifies of errors like file system permission issues in the top right messages dropdown.

Since I rarely use Splunk web interface, I'm always missing them.

Is there a way to get notified of these errors? Can I set up alerts for them?

0 Karma
1 Solution

gjanders
SplunkTrust
SplunkTrust

I've built a library of alerts into an app called "Alerts For Splunk Admins (on SplunkBase)" or github link , various alerts address a large number of potential "ERROR" scenarios in Splunk.

The app was mostly built by looking for ERROR/WARNING in the logs and identifying which ones caused a system or user impact of some kind, however if you want to look for just error/warning in the internal logs you likely want to run something like:

index=_internal sourcetype=splunkd source=*splunkd.log ERROR OR WARN | cluster show_count=true

And then you get a giant list of entries that mention error or warning, you may also want to watch the scheduler.log and the mongod.log, however in mongodb you look for " E " OR " F "

View solution in original post

gjanders
SplunkTrust
SplunkTrust

I've built a library of alerts into an app called "Alerts For Splunk Admins (on SplunkBase)" or github link , various alerts address a large number of potential "ERROR" scenarios in Splunk.

The app was mostly built by looking for ERROR/WARNING in the logs and identifying which ones caused a system or user impact of some kind, however if you want to look for just error/warning in the internal logs you likely want to run something like:

index=_internal sourcetype=splunkd source=*splunkd.log ERROR OR WARN | cluster show_count=true

And then you get a giant list of entries that mention error or warning, you may also want to watch the scheduler.log and the mongod.log, however in mongodb you look for " E " OR " F "

reallyliri
Explorer

Thanks, this looks great! Would expect Splunk to do such a basically needed feature themselves.

0 Karma

gjanders
SplunkTrust
SplunkTrust

Ideally I'd like to see the majority of the alerts I created built into Splunk in some form (preferably with a 'notify me by email button')!

To be fair, some of them such as when you have buckets rolling too fast or similar are now shown in the console in 7.2 which is great, previously you had to be watching the logs to know about this...however I'd like to see a lot more built into Splunk

Anyway hopefully it helps, good luck!

0 Karma

guarisma
Contributor

You can look for internal errors with something like

index=_internal log_level=ERROR

And produce Alerts for the results.

0 Karma

lakshman239
SplunkTrust
SplunkTrust

The banner messages are available in logs and can be seen using

|rest /services/messages

You can create an alert based on what you need from the above to alert for new messages.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...