Splunk Search

Does anyone know what the metric "active_searches" in remote_searches.log represents?

chris
Motivator

Does anyone know what the metric 'active_searches' in remote_searches.log represents?

This is a sample log event:

04-02-2015 10:50:26.078 +0200 INFO  StreamedSearch - Streamed search connection established: server=indexer04, active_searches=53

I'm assuming that this is the total number of currently active searches (real time, scheduled and ad-hoc searches) that are running on the system that creates the log.

Is this metric a good indicator to show that a Splunk installation is saturated?

e. g. A constant value around 50 is not a good value for a 24 cpu core indexers since one search takes up one cpu core?

Regards
Chris
Ps:
This search from the S.o.S App only shows a couple of skipped and deferred searches every hour so the searches do get executed, but the cpu load on the indexers sometimes goes up to almost 100% for a couple of seconds (using top/sar) the average load is 50%.

index=_internal host="searchhead" source=*metrics.log group=searchscheduler
| timechart partial=false sum(dispatched) AS Started, sum(skipped) AS Skipped
| appendcols [search `set_internal_index` host="splunk01" sourcetype=scheduler status=continued
| eval savedsearch_id_scheduled_time=savedsearch_id."-".scheduled_time
| timechart dc(savedsearch_id_scheduled_time) AS Deferred]
1 Solution

apilger_splunk
Splunk Employee
Splunk Employee

Is the # of concurrent searches on that peer at the time the job was run. Yes, you can use this # to determine the search concurrency at a given point in time on each search peer.
It is only one indicator for what is going on your systems.

/alex

View solution in original post

apilger_splunk
Splunk Employee
Splunk Employee

Is the # of concurrent searches on that peer at the time the job was run. Yes, you can use this # to determine the search concurrency at a given point in time on each search peer.
It is only one indicator for what is going on your systems.

/alex

Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...