Archive
Highlighted

Send splunk internal warning and alerts via eMail

Motivator

How can i send Splunk internal alarms like "indexer queue full, skipping internal indexes" etc to a eMail recipient?

Tags (2)
0 Karma
Highlighted

Re: Send splunk internal warning and alerts via eMail

SplunkTrust
SplunkTrust

Two options, you can use the internal sendemail command or the sendresults add-on for Splunk which provides much functionality and flexibility than the sendemail command.

Sendresults: https://splunkbase.splunk.com/app/1794/

Sendemail: http://docs.splunk.com/Documentation/Splunk/6.2.3/SearchReference/Sendemail

0 Karma
Highlighted

Re: Send splunk internal warning and alerts via eMail

Motivator

Hi @joshd,
thanks i am aware of this command. What i am unaware of is how to search for - um, banner messages and such internal errors...

0 Karma
Highlighted

Re: Send splunk internal warning and alerts via eMail

Influencer

I wonder if the REST API could help here.. Specifically /services/messages

I'm away from a Splunk instance currently to try things out and confirm but I would think that a query like:

| rest /services/messages count=0

Would work and then standard alerting from it... Or maybe you'd have to use map to loop over the subresources too but it might be a start 🙂

0 Karma
Highlighted

Re: Send splunk internal warning and alerts via eMail

Influencer

That said however, if the warning is about min disk space being reached a Splunk search would not be kicked off either (assuming the volume with the dispatch directory is the issue), so maybe monitoring with another tool like Zabbix could be warranted as well

Highlighted

Re: Send splunk internal warning and alerts via eMail

Motivator

Thanks, i will look into it, it certainly looks promissing.

For the second part we use Nimsoft Nimbus and monitor the logfile of the forwarder and react to "ERRORconnectionfailed".

0 Karma
Highlighted

Re: Send splunk internal warning and alerts via eMail

Motivator
0 Karma