Alerting for splunkd

[Ant1D]

Hey,

I have seen the following messages in the past when using/trying to access the Splunk UI:

Splunkd daemon is not responding...

The splunkd daemon cannot be reached by splunkweb

I would like to set up some alerts to inform me when the above events happen. How can this be done?
Does anyone have a ready-made search query for this? Is it simply the case of building search queries that look for such phrases in the internal logs (splunkd.log)?

Thanks in advance for your help.

[mhassan]

Here is a simple script that test for port 8089 connectivity. It will send email alert (and restart the process if not connection):

/export/home/mhassan/scripts $ cat splunkd-watchdog.sh

!/bin/bash

splunkd-watchdog.sh

Monitor Splunk port 8089. Restart process if no answer

-Mohamad Hassan 2/2/215

timestamp=date
USER=whoami
WHO=who -u
host=uname -n

LOGFILE="/opt/splunk/var/log/splunk-watchdog.log"
PROC="splunkd"
PORT="8089"
ADMINS="user1@domain.com user2@domain.com"
TEST=nc -z -w5 $HOSTNAME $PORT;echo $? < /dev/null

Debugging

let TEST=0

echo $TEST

if [ $TEST -eq 1 ] ;then

    /opt/splunk/bin/splunk restart
    RESULT=`ps ax | grep -i "$PROC" | grep -v grep|grep -v watchdog`

    echo  "$timestamp $HOSTNAME: splunk-watchdog ALERT! $PROC process restarted" >> $LOGFILE

    echo -e "Splunk watchdog ALERT! Splunk port $PORT is not answering.\n\n Hostname:    $HOSTNAME\n Time:               $timestamp\n Notifying:      $ADMINS\n ACTION:       Restarting ($PROC)!\n\nCurrrent logged-in users:\n$WHO\n\nCurrent splunkd status(after restart):\n$RESULT" |mail -s "$PROC watchdog ALERT! [$HOSTNAME]"  $ADMINS

    echo "$PROC restarted!"
    echo "Email sent to: $ADMINS"

fi

[wrangler2x]

I don't know what sort of output you'd like with this search but this would do the job by providing a count that you can alert on (count > 0).

index=_internal source="/opt/splunk/var/log/splunk/web_service.log" log_level="error" sourcetype="splunk_web_service"  "Splunkd daemon is not responding" | stats earliest(_time) as FirstOccurance, latest(_time) as LastOccurance count |eval FirstOccurance=strftime(FirstOccurance,"%m/%d/%Y %H:%M:%S")|eval LastOccurance=strftime(LastOccurance,"%m/%d/%Y  %H:%M:%S")

If you don't have an occurrence of this in your current splunkd.log you can change the "source" part to end with a '*' and it will look in all the rotated-out logs as well as the current. Then just set the time window to go back to when you had one of these.