Send splunk internal warning and alerts via eMail

[dominiquevocat]

How can i send Splunk internal alarms like "indexer queue full, skipping internal indexes" etc to a eMail recipient?

[acharlieh]

I wonder if the REST API could help here.. Specifically /services/messages

I'm away from a Splunk instance currently to try things out and confirm but I would think that a query like:

| rest /services/messages count=0

Would work and then standard alerting from it... Or maybe you'd have to use map to loop over the subresources too but it might be a start 🙂

[acharlieh]

That said however, if the warning is about min disk space being reached a Splunk search would not be kicked off either (assuming the volume with the dispatch directory is the issue), so maybe monitoring with another tool like Zabbix could be warranted as well

[dominiquevocat]

Someone else came up with this (only saw it now: http://answers.splunk.com/answers/148023/how-to-convert-a-bulletin-message-to-an-alert.html )

[dominiquevocat]

Thanks, i will look into it, it certainly looks promissing.

For the second part we use Nimsoft Nimbus and monitor the logfile of the forwarder and react to "ERROR*connection*failed".

[joshd]

Two options, you can use the internal sendemail command or the sendresults add-on for Splunk which provides much functionality and flexibility than the sendemail command.

Sendresults: https://splunkbase.splunk.com/app/1794/

Sendemail: http://docs.splunk.com/Documentation/Splunk/6.2.3/SearchReference/Sendemail

[dominiquevocat]

Hi @joshd,
thanks i am aware of this command. What i am unaware of is how to search for - um, banner messages and such internal errors...