Monitoring Splunk

Reasonable Search performance?

lee28
New Member

Hi,
We ran a search command(just count the total event) and got the following results. (using 3 indexers)
total event count = 82,843,934

duration = 2,413.578 sec

Is it reasonable? looks to me that the search speed is quite slow.
Is there any way to increase the search performance?
Changing the settings in 'limits.conf or 'times.conf' file will help?

Thanks in advance
Julian

Tags (1)
0 Karma

dwaddle
SplunkTrust
SplunkTrust

Well, "reasonable" is sometimes subjective. Just doing some basic maths here -- 82,843,934 events / 2,413 secs = 34,332 events per second scan rate. If you divide that by the number of indexers (assuming the data is perfectly distributed, which may not be true) that is 11,444 events per second per indexer.

Another assumption that each event is 1000 bytes (which may not be true) puts your throughput around 11 MBytes/sec - which is low relative to the basic throughput of a modern disk subsystem. You do have an appropriate disk subsystem attached, right? And these are physical machines, or VMs?

This also includes overhead from search-head to indexer coordination, CPU-time cost of doing field extraction, and a few other things. You really don't have the information to see where all the time was spent. There's a search job inspector tool that can help. Perhaps you can update with data from it?

But, I think there is a bit of misconception here. A search to "count ALL the things!" is not really a objective test of search performance. You need to search for something other than "everything". A highly dense search (where the number of events returned is a large fraction of the total number of events in the system) will usually be slower than a relatively sparse one.

Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Manage Edge Processor pipelines end to end with APIs

This guide uses the Data Management APIs to create an Edge Processor, onboard one instance, deploy a pipeline ...

Splunk Brings Proof of Control to Digital Asset Transfers

Digital assets now sit inside the core of financial services. HSBC launched its Tokenized Deposit Service in ...

Manage Ingest Processor pipelines end to end with APIs

This guide uses the Data Management APIs to create an Ingest Processor pipeline, deploy it to the ...