Deployment Architecture
Provide Splunk Cloud feedback in this confidential UX survey by June 17
for a chance to win a $200 Amazon gift card!

How can I find duplicate scheduled searches running in a search head clustering environment?

sat94541
Communicator

I have a three Node Search Head Cluster environment and I suspect that some of the scheduled searches are running multiple times. How can I find these duplicate Scheduled searches running in an SHC environment.

1 Solution

rbal_splunk
Splunk Employee
Splunk Employee

If you have a DMC that searches across all SHC member on Scheduler.log - you can proceed with the following steps to debug, or else you need to index Scheduler.log from all SHC members somewhere.

1)Run the search below to check if any scheduled search was run multiple times:

( host=  OR host=) )  source=*scheduler.log  status=success 
| rex field=sid "(?\w+_\w+_\w+_\w+_at_\d+)_" 
| eval secDiff=dispatch_time-scheduled_time 
| eval schedT=strftime(scheduled_time, "%F %H:%M:%S") 
| eval dispatchT=strftime(dispatch_time, "%F %H:%M:%S") 
| transaction shortID keepevicted=t 
| search linecount>1
|table savedsearch_name, sid, schedT, dispatchT, secDiff, alert_actions

Sample Result:

alt text

2) Next you can focus on duplicate sid of one saved search like below to get more detail on the sid like and use the search below to get more details

 (
  host=  OR host=
 )
 AND
 (
  (source=*scheduler.log*   AND status=success   AND ( sid=scheduler__admin_dWlfc21zcw__RMD5ae47099b8f1c50d5_at_1460388900_111_96EB1F29-E71E-49E0-982C-767B6E64BE32 OR sid=scheduler__admin_dWlfc21zcw__RMD5ae47099b8f1c50d5_at_1460388900_122_181ABE0B-D122-42D2-A0C1-BACD9B46F50A ))
 OR 
 ( source=*splunkd.log* AND "Making node the captain" )

) | table _raw host

Result: In this case it shows duplicate were caused by the captain switch.

alt text

In this case duplicate were caused due to switch Captain- this is from Splunk version 6.2.6.

View solution in original post

rbal_splunk
Splunk Employee
Splunk Employee

If you have a DMC that searches across all SHC member on Scheduler.log - you can proceed with the following steps to debug, or else you need to index Scheduler.log from all SHC members somewhere.

1)Run the search below to check if any scheduled search was run multiple times:

( host=  OR host=) )  source=*scheduler.log  status=success 
| rex field=sid "(?\w+_\w+_\w+_\w+_at_\d+)_" 
| eval secDiff=dispatch_time-scheduled_time 
| eval schedT=strftime(scheduled_time, "%F %H:%M:%S") 
| eval dispatchT=strftime(dispatch_time, "%F %H:%M:%S") 
| transaction shortID keepevicted=t 
| search linecount>1
|table savedsearch_name, sid, schedT, dispatchT, secDiff, alert_actions

Sample Result:

alt text

2) Next you can focus on duplicate sid of one saved search like below to get more detail on the sid like and use the search below to get more details

 (
  host=  OR host=
 )
 AND
 (
  (source=*scheduler.log*   AND status=success   AND ( sid=scheduler__admin_dWlfc21zcw__RMD5ae47099b8f1c50d5_at_1460388900_111_96EB1F29-E71E-49E0-982C-767B6E64BE32 OR sid=scheduler__admin_dWlfc21zcw__RMD5ae47099b8f1c50d5_at_1460388900_122_181ABE0B-D122-42D2-A0C1-BACD9B46F50A ))
 OR 
 ( source=*splunkd.log* AND "Making node the captain" )

) | table _raw host

Result: In this case it shows duplicate were caused by the captain switch.

alt text

In this case duplicate were caused due to switch Captain- this is from Splunk version 6.2.6.

View solution in original post

Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!