Monitoring Splunk

Reasonable Search performance?

lee28
New Member

Hi,
We ran a search command(just count the total event) and got the following results. (using 3 indexers)
total event count = 82,843,934

duration = 2,413.578 sec

Is it reasonable? looks to me that the search speed is quite slow.
Is there any way to increase the search performance?
Changing the settings in 'limits.conf or 'times.conf' file will help?

Thanks in advance
Julian

Tags (1)
0 Karma

dwaddle
SplunkTrust
SplunkTrust

Well, "reasonable" is sometimes subjective. Just doing some basic maths here -- 82,843,934 events / 2,413 secs = 34,332 events per second scan rate. If you divide that by the number of indexers (assuming the data is perfectly distributed, which may not be true) that is 11,444 events per second per indexer.

Another assumption that each event is 1000 bytes (which may not be true) puts your throughput around 11 MBytes/sec - which is low relative to the basic throughput of a modern disk subsystem. You do have an appropriate disk subsystem attached, right? And these are physical machines, or VMs?

This also includes overhead from search-head to indexer coordination, CPU-time cost of doing field extraction, and a few other things. You really don't have the information to see where all the time was spent. There's a search job inspector tool that can help. Perhaps you can update with data from it?

But, I think there is a bit of misconception here. A search to "count ALL the things!" is not really a objective test of search performance. You need to search for something other than "everything". A highly dense search (where the number of events returned is a large fraction of the total number of events in the system) will usually be slower than a relatively sparse one.

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...