topic Re: Alerting for splunkd in Alerting

Alerting for splunkd

Ant1D — Mon, 03 Dec 2012 15:33:26 GMT

Hey,

I have seen the following messages in the past when using/trying to access the Splunk UI:

Splunkd daemon is not responding...

The splunkd daemon cannot be reached by splunkweb

I would like to set up some alerts to inform me when the above events happen. How can this be done?
Does anyone have a ready-made search query for this? Is it simply the case of building search queries that look for such phrases in the internal logs (splunkd.log)?

Thanks in advance for your help.

Re: Alerting for splunkd

wrangler2x — Wed, 19 Feb 2014 23:52:04 GMT

I don't know what sort of output you'd like with this search but this would do the job by providing a count that you can alert on (count > 0).

index=_internal source="/opt/splunk/var/log/splunk/web_service.log" log_level="error" sourcetype="splunk_web_service"  "Splunkd daemon is not responding" | stats earliest(_time) as FirstOccurance, latest(_time) as LastOccurance count |eval FirstOccurance=strftime(FirstOccurance,"%m/%d/%Y %H:%M:%S")|eval LastOccurance=strftime(LastOccurance,"%m/%d/%Y  %H:%M:%S")

If you don't have an occurrence of this in your current splunkd.log you can change the "source" part to end with a '*' and it will look in all the rotated-out logs as well as the current. Then just set the time window to go back to when you had one of these.

Re: Alerting for splunkd

mhassan — Wed, 25 Feb 2015 17:36:35 GMT

Here is a simple script that test for port 8089 connectivity. It will send email alert (and restart the process if not connection):

/export/home/mhassan/scripts $ cat splunkd-watchdog.sh

!/bin/bash

splunkd-watchdog.sh

Monitor Splunk port 8089. Restart process if no answer

-Mohamad Hassan 2/2/215

timestamp=date
USER=whoami
WHO=who -u
host=uname -n

LOGFILE="/opt/splunk/var/log/splunk-watchdog.log"
PROC="splunkd"
PORT="8089"
ADMINS="user1@domain.com user2@domain.com"
TEST=nc -z -w5 $HOSTNAME $PORT;echo $? < /dev/null

Debugging

let TEST=0

echo $TEST

if [ $TEST -eq 1 ] ;then

    /opt/splunk/bin/splunk restart
    RESULT=`ps ax | grep -i "$PROC" | grep -v grep|grep -v watchdog`

    echo  "$timestamp $HOSTNAME: splunk-watchdog ALERT! $PROC process restarted" >> $LOGFILE

    echo -e "Splunk watchdog ALERT! Splunk port $PORT is not answering.\n\n Hostname:    $HOSTNAME\n Time:               $timestamp\n Notifying:      $ADMINS\n ACTION:       Restarting ($PROC)!\n\nCurrrent logged-in users:\n$WHO\n\nCurrent splunkd status(after restart):\n$RESULT" |mail -s "$PROC watchdog ALERT! [$HOSTNAME]"  $ADMINS

    echo "$PROC restarted!"
    echo "Email sent to: $ADMINS"