Monitoring Splunk

Splunkd Health Status - Search Scheduler - Search Lag & Delay

anirbandasdeb
Path Finder

Hello all, On the splunkd health report, what is the difference between Search Lag & Delay? [ref: https://docs.splunk.com/images/e/ee/Splunkd_health_report_8.0.0.png]

Our deployment has a high number of savedsearches that trigger on cron (every 5m, 15m, 30m, 1h etc) and we are working to minimise the concurrency by introducing Scheduler Window & Skew.
I know exactly which searches are triggering beyond the scheduled time (dispatch_time - scheduled_time from the scheduler.log) and which searches are skipping.
But I do not understand what Splunk signifies as Lag & Delay in terms of searches..

I have gone through the $SPLUNK_HOME/var/log/health.log & server/health/splunkd/details API endpoints but they give the same messages as the Health Indicator.. Thanks in advance!

0 Karma

janroc
Explorer

Hi,

There is an app on splunkbase named Alerts For Splunk Admins.

Have you tried the app to find the slowness?

 

Regards Jan

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...