In Splunk Enterprise when looking at the metrics.log with the searchscheduler group there is a metric for "eligible" but I can't find out what this indicates.
index=_internal source=*metrics.log group=searchscheduler
For context this is on Splunk Enterprise 8.0.2006 Cloud and we have 3 search heads in a cluster.
I was able to find this documentation but in the section where it talks about groups there is nothing mentioning the searchscheduler group
Here's an example event:
09-29-2020 19:06:28.281 +0000 INFO Metrics - group=searchscheduler, eligible=9, delayed=0, dispatched=0, skipped=0, total_lag=0, max_lag=0, window_max_lag=0, window_total_lag=0, max_running=3, actions_triggered=0, completed=3, total_runtime=21.251, max_runtime=19.212
The reason I am interested in this value is that when looking at all of the other metrics in this group (delegated, delegated_scheduled, delegated_waiting, dispatched, eligible, skipped, delayed, completed, actions_triggered) there was a noticeable dip in only the "eligible" metric during some periods where our alerts were not triggering actions. The dip in this metric affected only 2 out of 3 search heads.
