Splunk Search

Forwarder not reporting more than 2 weeks ago.

sanju005ind
Communicator

I need help with a query to find the forwarders which stopped reporting for more than 2 weeks.

Tags (1)
0 Karma
1 Solution

thall79
Communicator

This is my search that I use track the last time a forwarder has responded.

| metadata type=hosts | eval age = now() - lastTime | sort -age | search age > 86400 | eval d=floor(age/86400) | eval h=floor((age-(d*86400))/3600) | eval m=floor((age-(d*86400)-(h*3600))/60) | eval s=floor(age-(d*86400)-(h*3600)-(m*60)) | eval age =d." days ".h." hours ".m." minutes ".s." seconds" | fields age,host | rename age as Time_of_Last_Response

travis.

View solution in original post

thall79
Communicator

This is my search that I use track the last time a forwarder has responded.

| metadata type=hosts | eval age = now() - lastTime | sort -age | search age > 86400 | eval d=floor(age/86400) | eval h=floor((age-(d*86400))/3600) | eval m=floor((age-(d*86400)-(h*3600))/60) | eval s=floor(age-(d*86400)-(h*3600)-(m*60)) | eval age =d." days ".h." hours ".m." minutes ".s." seconds" | fields age,host | rename age as Time_of_Last_Response

travis.

sanju005ind
Communicator

Thanks.This is what I was looking for.

0 Karma

rotten
Communicator

I check my forwarder status with a shell script. That way I can check the indexer status also, and if I have a system which doesn't send logs very often, I can still make sure the forwarder is alive.

All of my forwarders on on Unix systems, so my script, invoked from cron every few minutes, does something like the snippet below.

I do take a short cut because all of my forwarder's PID files are available via NFS. However, you should be able to get the idea:


for splunker in `cat my_list_of_splunk_forwarders`
do

  PIDList=`cat ${splunker}/var/run/splunk/splunkd.pid`
  PIDCount=`print ${PIDList} | wc -w`

  SplunkFwdr=`print ${splunker} | cut -d- -f1`

  SplunkdCounter=0
  for PID in `print ${PIDList}`
  do
     ProcName=`ssh ${SplunkFwdr} ps -o comm -p "${PID}" | tail +2`

     if [[ ${ProcName} == "splunkd" ]]
     then

       SplunkdCounter=$((SplunkdCounter + 1))


     fi

  done

  ## There should always be at least two processes running:
  if [[ ${SplunkdCounter} -lt 2 ]]
  then
      print "${splunker} -- splunkd count:  ${SplunkdCounter}" | mailx -s "[${splunker}][Splunk] Forwarder Co
unt Mismatch [Found: ${SplunkdCounter}][Expected:  ${PIDCount}]" ${ToList}
  fi

done
0 Karma

Chris_R_
Splunk Employee
Splunk Employee

If your on version 4 try a query such as this to see recently connected forwarders.

| metadata type=hosts | sort recentTime desc | convert ctime(recentTime) as Recent_Time
0 Karma
Get Updates on the Splunk Community!

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...