Splunk Search

Scheduled reports working inconsistently.

sajeeshpn
New Member

What could be the reasons why scheduled reports are not getting executed all the time ?

We get log messages like:-
06-05-2017 13:21:27.015 +0000 WARN DispatchReaper - Too many search jobs found in the dispatch directory (found=3069, warning level=3000). This could negatively impact search starting times
06-05-2017 13:26:27.017 +0000 WARN DispatchReaper - Too many search jobs found in the dispatch directory (found=3069, warning level=3000). This could negatively impact search starting times
06-05-2017 13:31:57.010 +0000 WARN DispatchReaper - Too many search jobs found in the dispatch directory (found=3069, warning level=3000). This could negatively impact search starting times
06-05-2017 13:36:57.014 +0000 WARN DispatchReaper - Too many search jobs found in the dispatch directory (found=3069, warning level=3000). This could negatively impact search starting times
06-05-2017 13:41:57.015 +0000 WARN DispatchReaper - Too many search jobs found in the dispatch directory (found=3069, warning level=3000). This could negatively impact search starting times

and also logs like "The maximum number of historical concurrent system-wide searches has been reached", quite regularly.

Will that be a reason why scheduled reports are not working or gets skipped occasionally ? Or are there any system resources constraints (like CPU) which is causing this issue.

Please note: We only have few scheduled reports configured.

Tags (1)
0 Karma

woodcock
Esteemed Legend

What @martin_mueller said but focus on All time and Real-time searches. You should completely disable Real-time searches system-wide and you should change the default from All time to Last 15 minutes. Train your users, too.

martin_mueller
SplunkTrust
SplunkTrust

The dispatch warnings tell you that lots of search artifacts are sitting in your dispatch directory. Not a problem per se, but can slow things down a bit as mentioned in the event.

The maximum number of concurrent system-wide searches can indeed cause scheduled reports to skip. If there are no slots to run the report for an entire period, scheduled jobs would pile up infinitely.
Add more faster CPUs. Add more boxes. Speed up existing inefficient searches. Clean up existing but not needed searches. For the last two, focus on long-running and frequently-running searches.

sajeeshpn
New Member

How can we get to know that any scheduled reports have been "SKIPPED" ? Do we have any specific Splunk logs to denote that ?

0 Karma
Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...