Can I send an email if splunk is not running

[terryjohn]

I'm running Splunk 6.2.3 and it sends alert emails as expected. Recently I had an issue where the indexer was stopped and inadvertently not restarted.

I have already implemented a cron job to restart splunk should this happen again but this can only work out of hours since in the normal working day splunk may be stopped for valid reasons.

I would like to write a script that checks Splunk is running and if not send an email using Splunk's standard settings and methods. This would save me re-inventing the wheel and writing my own email sender. In addition it would use Splunk's current email settings which are always kept current.

I'm just wondering if there is a recommended way of sending emails this way.

[grijhwani]

cron this:

ps -ef | grep -q splunk[d] || { mailx -s "no splunk detected" your@address </dev/null >/dev/null; }

It will check the process list for splunkd processes and email a short, subject-only message if none are found. (I'm assuming you're using Linux, given that you already mentioned cron yourself.)

[terryjohn]

Yes Centos 6. Unfortunately, for security reasons, we have no mail transfer agent listening on port 25. To be honest it's a mystery to me how Splunk sends emails. It has the required email user and password configured. I need access to these settings.

[grijhwani]

There's no mystery. Splunk is using its own embedded Python module to create its own session with your local mail relay, rather than relying on a local MTA. There is nothing stopping you doing the same.

This article addresses the problem specifically on Ubuntu, but the method is completely portable. It may require that you install a system-wide deployment of Python. Trying to use Splunk's embedded Python from a cron job will be a hiding to nothing. Bear in mind though that anything you do to by-pass the need for an MTA is also shooting holes in your security policy, because anything available to your script will be available to an intruder.

[terryjohn]

Thanks for that, really useful. Without adding or changing anything I've got a version of that running so I can send a warning email. So we're getting somewhere.

Specifically I was asking on this forum to see if I could gain access to the Splunk Email settings in Settings->Server Settings->Email Settings.

If I could read that then any changes in the Splunk email settings would immediately be available to this script.

[terryjohn]

Ha. Found the file
$SPLUNK_HOME/etc/system/local/alert_actions.conf
Lost of good stuff in there but the auth_password is encrypted.
Frustrating. I know what the password is, but the script needs to read it when the password gets changed next time
So close.. so far

[bohrasaurabh]

When you run the below search, how many splunk_server do you get in your results.

index=_internal | stats count by splunk_server

[terryjohn]

Just the one

[bohrasaurabh]

I have an Alert setup which runs every 30 mins and looks for 5 or more instances of "Connect to *:9997 failed" if host is any of my searchheads. If found sends email to Admins.

index="_internal" source="*splunkd.log" host="SEARCH_HEAD_NAME*" "Connect to *:9997 failed"

[terryjohn]

Unfortunately that would not help as Splunk indexer itself would not be running even though the server is up.

[bohrasaurabh]

I should have mentioned that it will only work if you have more than 1 index server. Based on your posting I assumed you have more that 1 index servers, how many do you have?

[terryjohn]

Yes. Only the 1 in this setup