Splunk Enterprise

Health Monitor Engine - Scheduler: Lags and Skipped

verbal_666
Builder

Hi there.
I would like to know about Splunk Health engine, Enterprise 8.2.12, 3 SHC,

 

verbal_666_1-1703216686894.png

 

  1. HOW it considers a savedsearch a Lagged search? Based on same previous 24h search runs and doing an average running times? Since we have many many heavy searches that end up also in 10/15m
  2. WHY, sometimes, i found in Skipped search monitor a 100% of skipped search (1 from 1, when we have hundreds of scheduled searches)? WHILE, searching the scheduler log, i found something like 70.000 success / 68 skipped (scheduled every minute or every two, concurrency is a factor i calculate and there's no problem) in last 24h ? WHY 100%? Is it a bug? I also search for a single scheduled search per day savedsearches, but all (few) are in "success" status 🙄

When those strange things occur, sometimes, restarting the cluster, make health monitor to reset without warnings!!! Other times, in reverse, restarting the cluster make a clean health monitor to start giving warnings from point 1 & 2 🙄 ... strange behaviour!!! 😒

Thanks.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

this could explain that behaviour to you https://docs.splunk.com/Documentation/Splunk/9.1.2/DMC/Configurefeaturemonitoring

Based on this instructions you could see what those health messages means.

r. Ismo

0 Karma
Get Updates on the Splunk Community!

Thanks for the Memories! Splunk University, .conf24, and Community Connections

Thank you to everyone in the Splunk Community who joined us for .conf24 – starting with Splunk University and ...

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...