Splunk Search

Does anyone know what the metric "active_searches" in remote_searches.log represents?

chris
Motivator

Does anyone know what the metric 'active_searches' in remote_searches.log represents?

This is a sample log event:

04-02-2015 10:50:26.078 +0200 INFO  StreamedSearch - Streamed search connection established: server=indexer04, active_searches=53

I'm assuming that this is the total number of currently active searches (real time, scheduled and ad-hoc searches) that are running on the system that creates the log.

Is this metric a good indicator to show that a Splunk installation is saturated?

e. g. A constant value around 50 is not a good value for a 24 cpu core indexers since one search takes up one cpu core?

Regards
Chris
Ps:
This search from the S.o.S App only shows a couple of skipped and deferred searches every hour so the searches do get executed, but the cpu load on the indexers sometimes goes up to almost 100% for a couple of seconds (using top/sar) the average load is 50%.

index=_internal host="searchhead" source=*metrics.log group=searchscheduler
| timechart partial=false sum(dispatched) AS Started, sum(skipped) AS Skipped
| appendcols [search `set_internal_index` host="splunk01" sourcetype=scheduler status=continued
| eval savedsearch_id_scheduled_time=savedsearch_id."-".scheduled_time
| timechart dc(savedsearch_id_scheduled_time) AS Deferred]
1 Solution

apilger_splunk
Splunk Employee
Splunk Employee

Is the # of concurrent searches on that peer at the time the job was run. Yes, you can use this # to determine the search concurrency at a given point in time on each search peer.
It is only one indicator for what is going on your systems.

/alex

View solution in original post

apilger_splunk
Splunk Employee
Splunk Employee

Is the # of concurrent searches on that peer at the time the job was run. Yes, you can use this # to determine the search concurrency at a given point in time on each search peer.
It is only one indicator for what is going on your systems.

/alex

Get Updates on the Splunk Community!

Get ready to show some Splunk Certification swagger at .conf24!

Dive into the deep end of data by earning a Splunk Certification at .conf24. We're enticing you again this ...

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Now On-Demand Join us to learn more about how you can leverage Service Level Objectives (SLOs) and the new ...

Database Performance Sidebar Panel Now on APM Database Query Performance & Service ...

We’ve streamlined the troubleshooting experience for database-related service issues by adding a database ...