Does anyone know what the metric 'active_searches' in remote_searches.log represents?
This is a sample log event:
04-02-2015 10:50:26.078 +0200 INFO StreamedSearch - Streamed search connection established: server=indexer04, active_searches=53
I'm assuming that this is the total number of currently active searches (real time, scheduled and ad-hoc searches) that are running on the system that creates the log.
Is this metric a good indicator to show that a Splunk installation is saturated?
e. g. A constant value around 50 is not a good value for a 24 cpu core indexers since one search takes up one cpu core?
Regards
Chris
Ps:
This search from the S.o.S App only shows a couple of skipped and deferred searches every hour so the searches do get executed, but the cpu load on the indexers sometimes goes up to almost 100% for a couple of seconds (using top/sar) the average load is 50%.
index=_internal host="searchhead" source=*metrics.log group=searchscheduler
| timechart partial=false sum(dispatched) AS Started, sum(skipped) AS Skipped
| appendcols [search `set_internal_index` host="splunk01" sourcetype=scheduler status=continued
| eval savedsearch_id_scheduled_time=savedsearch_id."-".scheduled_time
| timechart dc(savedsearch_id_scheduled_time) AS Deferred]
