Splunk Search

Does anyone know what the metric "active_searches" in remote_searches.log represents?

chris
Motivator

Does anyone know what the metric 'active_searches' in remote_searches.log represents?

This is a sample log event:

04-02-2015 10:50:26.078 +0200 INFO  StreamedSearch - Streamed search connection established: server=indexer04, active_searches=53

I'm assuming that this is the total number of currently active searches (real time, scheduled and ad-hoc searches) that are running on the system that creates the log.

Is this metric a good indicator to show that a Splunk installation is saturated?

e. g. A constant value around 50 is not a good value for a 24 cpu core indexers since one search takes up one cpu core?

Regards
Chris
Ps:
This search from the S.o.S App only shows a couple of skipped and deferred searches every hour so the searches do get executed, but the cpu load on the indexers sometimes goes up to almost 100% for a couple of seconds (using top/sar) the average load is 50%.

index=_internal host="searchhead" source=*metrics.log group=searchscheduler
| timechart partial=false sum(dispatched) AS Started, sum(skipped) AS Skipped
| appendcols [search `set_internal_index` host="splunk01" sourcetype=scheduler status=continued
| eval savedsearch_id_scheduled_time=savedsearch_id."-".scheduled_time
| timechart dc(savedsearch_id_scheduled_time) AS Deferred]
1 Solution

apilger_splunk
Splunk Employee
Splunk Employee

Is the # of concurrent searches on that peer at the time the job was run. Yes, you can use this # to determine the search concurrency at a given point in time on each search peer.
It is only one indicator for what is going on your systems.

/alex

View solution in original post

apilger_splunk
Splunk Employee
Splunk Employee

Is the # of concurrent searches on that peer at the time the job was run. Yes, you can use this # to determine the search concurrency at a given point in time on each search peer.
It is only one indicator for what is going on your systems.

/alex

Get Updates on the Splunk Community!

AppDynamics Summer Webinars

This summer, our mighty AppDynamics team is cooking up some delicious content on YouTube Live to satiate your ...

SOCin’ it to you at Splunk University

Splunk University is expanding its instructor-led learning portfolio with dedicated Security tracks at .conf25 ...

Credit Card Data Protection & PCI Compliance with Splunk Edge Processor

Organizations handling credit card transactions know that PCI DSS compliance is both critical and complex. The ...