topic Re: How to improve the speed of Spunk search in Splunk Search

How to improve the speed of Splunk search

qazwsxe — Thu, 30 Jan 2025 22:57:31 GMT

I want to get hundreds of millions of data from billions of data, but it takes more than an hour each time.
I just used the simplest search: index="test" name=jack But, it's very slow.

Then I checked the memory and CPU usage. Each search takes only 200-300 MB of memory.
So I modified the max_mem_usage_mb, search_process_memory_usage_percentage_threshold and search_process_memory_usage_threshold parameters in $SPLUNK_HOME/etc/apps/search/local/limits.conf, but they didn't seem to play a significant role.
Is there any effective way to improve the speed of my search?
Thanks! 🙂

Re: How to improve the speed of Spunk search

niketn — Fri, 28 Jun 2019 03:08:27 GMT

@qazwsxe one of the best way to index and monitor KPI information would be to use Metrics Index available from version 7.x. With each release there is significant new features introduced to Metrics Indexing, so do explore the latest version and features (like current latest version 7.3.0 introduces Metrics Rollup). https://docs.splunk.com/Documentation/Splunk/latest/Metrics/Overview

https://www.splunk.com/blog/2018/05/16/metrics-to-the-max-dramatic-performance-improvements-for-monitoring-and-alerting-on-metrics-data.html

https://www.splunk.com/blog/2019/06/18/navigating-data-chaos-with-splunk-metrics-workspace.html

Re: How to improve the speed of Spunk search

qazwsxe — Fri, 28 Jun 2019 07:24:21 GMT

@martin_mueller♦

Re: How to improve the speed of Spunk search

amitm05 — Fri, 28 Jun 2019 07:32:10 GMT

@ qazwsxe
Simplest searches are the most vague searches as well sometimes. For best performances below are a few suggestions you can use to optimize the performance of your query.

Be precise with your base search as much as you can. It helps reduce the scope of search. Great help in case of larger indexes. So Include everything you know about the events you want.
Adding a sourcetype after your index will help narrowing the scope of searched events i.e. like

`index="test" sourcetype=<yourST> name=jack`

Keep your time range precise about your requirement.
If you working in an indexer cluster. Make sure your index data is being split across all indexers. This will help avoid the load on single indexer instance for a search.

And Lastly, do take a look at the job inspector for your searches and analyse where the most time is getting spent. They will be the areas to work upon.

Thanks.

Re: How to improve the speed of Spunk search

qazwsxe — Fri, 28 Jun 2019 07:54:51 GMT

Even if the search conditions are accurate, the speed is slow.There's a lot of data, maybe billions,so the search speed is slow.
When the search command is executed, memory and CPU usage are minimal.Using the same data, ELK takes up a lot of memory and cpu, and it's much faster than splunk.
So, what should I do?
Thanks.

Re: How to improve the speed of Spunk search

martin_mueller — Fri, 28 Jun 2019 12:50:26 GMT

Be more specific in describing what goal you want your search to achieve. I doubt it's "list millions of events on screen" because there's no value in that.

Re: How to improve the speed of Spunk search

mbagali_splunk — Fri, 28 Jun 2019 13:00:44 GMT

@qazwsxe

For Faster Search, you need to be specific. Use source or sourcetype in your search and make use of Time range picker to search the logs in the specific time range.

Also, we have below modes of searching in Splunk:

Fast Mode
Smart Mode
Verbose Mode

Re: How to improve the speed of Spunk search

qazwsxe — Mon, 01 Jul 2019 01:50:28 GMT

I want to extract hundreds of millions of data from billions of data by simple keyword search, but the speed is too slow. No matter how much data is searched, CPU and memory usage have not changed significantly. Excuse me, is there something wrong with my usage? I just want to speed up my search.

Re: How to improve the speed of Spunk search

qazwsxe — Mon, 01 Jul 2019 02:22:16 GMT

I also used sourcetype and specified fast mode. But the speed is still very slow. I don't know how to solve it.

Re: How to improve the speed of Spunk search

qazwsxe — Mon, 01 Jul 2019 03:50:28 GMT

@adonio♦ Could you help me ?
Thanks 🙂

Re: How to improve the speed of Spunk search

martin_mueller — Mon, 01 Jul 2019 07:23:03 GMT

Okay, what do you want to do with those hundreds of millions of data?

Re: How to improve the speed of Spunk search

rvany — Mon, 01 Jul 2019 08:04:38 GMT

So what's your environment (regarding Indexer(s) and searchhead(s), storage)? And where are you examining the resource consumption?

Re: How to improve the speed of Spunk search

codebuilder — Mon, 01 Jul 2019 15:00:39 GMT

Increase the resources available to Splunk at the search head level.
Modify the settings below (based on your environment) at $SPLUNK_HOME:/etc/system/local/limits.conf and cycle the search head(s).

[defaults]
max_mem_usage_mb = 16000

[search]
# If number of cpu's in your machine is 14 then total system wide number of
# concurrent searches this machine can handle is 20.
# which is base_max_searches + max_searches_per_cpu x num_cpus = 6 + 14 x 1 = 20
base_max_searches = 6
max_searches_per_cpu = 16

[scheduler]
# Percent of total concurrent searches that will be used by scheduler is
# total concurrency x max_searches_perc = 20 x 60% = 12 scheduled searches
# User default value (needed only if different from system/default value) when
# no max_searches_perc.<n>.when (if any) below matches.
max_searches_perc = 80

Re: How to improve the speed of Spunk search

martin_mueller — Mon, 01 Jul 2019 15:05:01 GMT

The search currently provided, index=foo field=value, does not consume SH memory at all. It's purely Indexer-unzip-rawdata induced CPU bound.

Re: How to improve the speed of Spunk search

codebuilder — Mon, 01 Jul 2019 15:14:53 GMT

All searches consume memory on the search head. Assuming you are not running your query directly from the indexer UI.

The indexers perform the work of the query then pass those results back to the SH for any additional parsing and display to the end user. Depending on the size of your search artifacts this can produce tremendous resource consumption on both sides.

Re: How to improve the speed of Spunk search

martin_mueller — Mon, 01 Jul 2019 15:17:41 GMT

The search currently provided does not do any additional work on the SH, it's all map and no significant reduce.

In order to help this person we first need to understand their goals, not throw around tons of deep-dive tuning.

Re: How to improve the speed of Spunk search

codebuilder — Mon, 01 Jul 2019 15:30:07 GMT

In order to help this person we should also not provide inaccurate information, such as the idea that searches do not consume memory on a search head.

Re: How to improve the speed of Spunk search

woodcock — Mon, 01 Jul 2019 15:31:03 GMT

Splunk is not an ETL tool (it is a Needle-in-the-haystack tool, not a Forklift-the-haystack tool). There is no way to make it perform acceptably when the final output is millions of rows/events. It can process billions down to millions, and millions down to hundreds or maybe thousands, but that's it. You need to figure out what you really need to do and probably you don't need the millions as the final output, but rather as an input to some other calculation which can probably be done in Splunk. Otherwise, use another, more appropriate tool. Splunk is not a part of the pipeline tool, it is an end of the pipeline tool, to give the final conclusions.

Re: How to improve the speed of Spunk search

martin_mueller — Mon, 01 Jul 2019 15:31:52 GMT

This search doesn't, there is no command running on the SH.
It'd be different if there was a high-cardinality stats, a transaction, etc.

While you're defining what information to provide, I wouldn't recommend recommending max_searches_per_cpu = 16. It's a good way to thrash your indexing tier.

Re: How to improve the speed of Spunk search

qazwsxe — Tue, 02 Jul 2019 01:25:18 GMT

Because of the large amount of data, the data I get through keyword search will have a lot of correlation, so I will extract a lot of data, but I can not add restrictions, abandon part of the data, which will lead to slow speed. I am eager to solve this problem.
Thanks:)