Splunk Search

Does anyone know what the metric "active_searches" in remote_searches.log represents?

chris
Motivator

Does anyone know what the metric 'active_searches' in remote_searches.log represents?

This is a sample log event:

04-02-2015 10:50:26.078 +0200 INFO  StreamedSearch - Streamed search connection established: server=indexer04, active_searches=53

I'm assuming that this is the total number of currently active searches (real time, scheduled and ad-hoc searches) that are running on the system that creates the log.

Is this metric a good indicator to show that a Splunk installation is saturated?

e. g. A constant value around 50 is not a good value for a 24 cpu core indexers since one search takes up one cpu core?

Regards
Chris
Ps:
This search from the S.o.S App only shows a couple of skipped and deferred searches every hour so the searches do get executed, but the cpu load on the indexers sometimes goes up to almost 100% for a couple of seconds (using top/sar) the average load is 50%.

index=_internal host="searchhead" source=*metrics.log group=searchscheduler
| timechart partial=false sum(dispatched) AS Started, sum(skipped) AS Skipped
| appendcols [search `set_internal_index` host="splunk01" sourcetype=scheduler status=continued
| eval savedsearch_id_scheduled_time=savedsearch_id."-".scheduled_time
| timechart dc(savedsearch_id_scheduled_time) AS Deferred]
1 Solution

apilger_splunk
Splunk Employee
Splunk Employee

Is the # of concurrent searches on that peer at the time the job was run. Yes, you can use this # to determine the search concurrency at a given point in time on each search peer.
It is only one indicator for what is going on your systems.

/alex

View solution in original post

apilger_splunk
Splunk Employee
Splunk Employee

Is the # of concurrent searches on that peer at the time the job was run. Yes, you can use this # to determine the search concurrency at a given point in time on each search peer.
It is only one indicator for what is going on your systems.

/alex

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...