Monitoring Splunk

Reasonable Search performance?

lee28
New Member

Hi,
We ran a search command(just count the total event) and got the following results. (using 3 indexers)
total event count = 82,843,934

duration = 2,413.578 sec

Is it reasonable? looks to me that the search speed is quite slow.
Is there any way to increase the search performance?
Changing the settings in 'limits.conf or 'times.conf' file will help?

Thanks in advance
Julian

Tags (1)
0 Karma

dwaddle
SplunkTrust
SplunkTrust

Well, "reasonable" is sometimes subjective. Just doing some basic maths here -- 82,843,934 events / 2,413 secs = 34,332 events per second scan rate. If you divide that by the number of indexers (assuming the data is perfectly distributed, which may not be true) that is 11,444 events per second per indexer.

Another assumption that each event is 1000 bytes (which may not be true) puts your throughput around 11 MBytes/sec - which is low relative to the basic throughput of a modern disk subsystem. You do have an appropriate disk subsystem attached, right? And these are physical machines, or VMs?

This also includes overhead from search-head to indexer coordination, CPU-time cost of doing field extraction, and a few other things. You really don't have the information to see where all the time was spent. There's a search job inspector tool that can help. Perhaps you can update with data from it?

But, I think there is a bit of misconception here. A search to "count ALL the things!" is not really a objective test of search performance. You need to search for something other than "everything". A highly dense search (where the number of events returned is a large fraction of the total number of events in the system) will usually be slower than a relatively sparse one.

Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Get Agentic with Splunk Lantern: Connect to Cisco Cloud Control, Transform ...

Splunk Lantern is Splunk’s customer success center that provides practical guidance from Splunk experts on key ...

July Community Events: Master ITSI 5.0 & Automate Splunk

Struggling with alert fatigue or feeling like you're spending more time on infrastructure maintenance than ...

New Release of Federated Search: Bringing Splunk Analytics to More of Your Data

Organizations today are generating more data than ever and storing it across cloud object stores, data lakes, ...