Splunk Search

Does anyone know what the metric "active_searches" in remote_searches.log represents?

chris
Motivator

Does anyone know what the metric 'active_searches' in remote_searches.log represents?

This is a sample log event:

04-02-2015 10:50:26.078 +0200 INFO  StreamedSearch - Streamed search connection established: server=indexer04, active_searches=53

I'm assuming that this is the total number of currently active searches (real time, scheduled and ad-hoc searches) that are running on the system that creates the log.

Is this metric a good indicator to show that a Splunk installation is saturated?

e. g. A constant value around 50 is not a good value for a 24 cpu core indexers since one search takes up one cpu core?

Regards
Chris
Ps:
This search from the S.o.S App only shows a couple of skipped and deferred searches every hour so the searches do get executed, but the cpu load on the indexers sometimes goes up to almost 100% for a couple of seconds (using top/sar) the average load is 50%.

index=_internal host="searchhead" source=*metrics.log group=searchscheduler
| timechart partial=false sum(dispatched) AS Started, sum(skipped) AS Skipped
| appendcols [search `set_internal_index` host="splunk01" sourcetype=scheduler status=continued
| eval savedsearch_id_scheduled_time=savedsearch_id."-".scheduled_time
| timechart dc(savedsearch_id_scheduled_time) AS Deferred]
1 Solution

apilger_splunk
Splunk Employee
Splunk Employee

Is the # of concurrent searches on that peer at the time the job was run. Yes, you can use this # to determine the search concurrency at a given point in time on each search peer.
It is only one indicator for what is going on your systems.

/alex

View solution in original post

apilger_splunk
Splunk Employee
Splunk Employee

Is the # of concurrent searches on that peer at the time the job was run. Yes, you can use this # to determine the search concurrency at a given point in time on each search peer.
It is only one indicator for what is going on your systems.

/alex

Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...