topic Can someone explain concurrent historical searches to me? in Splunk Search

Can someone explain concurrent historical searches to me?

sboogaar — Tue, 29 Sep 2020 23:03:28 GMT

I do not understand what is meant by concurrent historical searches.

Can someone else explain what it means to me?

I read the entire Splunk limits.conf, where you can specify it, but I think i missed the explanation of what a concurrent historical search actually is.

Which searches are counted? subsearches/joins?

Is it possible to view the number for a specific search?

The calculation in the docs says:

max_hist_searches = max_searches_per_cpu x number_of_cpus + base_max_searches
So does the number of cores in your CPU not matter? We have 4 searchheads with 8 cores, so by default, we can do:

1 * 4 + 6  = 10

concurrent historical searches with those machines?

For each machine we add we get only 1 concurrent historical search extra?

Re: Can someone explain concurrent historical searches to me?

vishaltaneja070 — Tue, 29 Sep 2020 23:03:31 GMT

Hello @sboogar

A historical search is a search that has a distinct time range, such as the past hour, the previous day, or "between 2 and 4 last Tuesday". Historical searches usually review data in the past, but you can set up these searches to review events with future-dated timestamps, if your index contains them.

In Limits.conf there are different parameter which can help you out. This is the formula which is used for this:

max_searches_per_cpu x number_of_cpus + base_max_searches = max_hist_searches

The default settings are base_max_searches = 6, max_searches_per_cpu = 1, max_searches_per = 50, and max_rt_search_multiplier = These settings can be changes using limits.conf but you need to be careful as it all depends on your hardware capability.

Re: Can someone explain concurrent historical searches to me?

sboogaar — Fri, 01 Feb 2019 12:54:20 GMT

@vishaltaneja07011993 Do subsearches/joins count? what if I have 5 joins in a query do I have 6 concurrent historical searches than?

Re: Can someone explain concurrent historical searches to me?

vishaltaneja070 — Fri, 01 Feb 2019 12:58:38 GMT

No, subsearch will run first and then the base search. But there is a catch: By default, subsearches return a maximum of 10,000 results and have a maximum runtime of 60 seconds. In large production environments, it is possible that the subsearch will timeout before it completes. The best option is to rewrite the query to limit the number of events that the subsearch must process.

Re: Can someone explain concurrent historical searches to me?

sboogaar — Fri, 01 Feb 2019 13:02:12 GMT

But even if all the joins return only 1 results it is counted as 6 concurrent historical searches right? so if I run 5 querys with 5 joins on the same time im doing 30 concurrent historical searches?

Re: Can someone explain concurrent historical searches to me?

vishaltaneja070 — Fri, 01 Feb 2019 13:07:57 GMT

yes it will count as 6 searches but not concurrent as it will run one by one not at a single which can cause a block. Thats why there is a max runtime and max search result limit on subsearch.

Re: Can someone explain concurrent historical searches to me?

sboogaar — Fri, 01 Feb 2019 13:22:37 GMT

@vishaltaneja07011993 I do not see the relation between blocking searches and " max runtime and max search result limit on subsearch."

Re: Can someone explain concurrent historical searches to me?

vishaltaneja070 — Fri, 01 Feb 2019 13:36:57 GMT

@sboogaar
Don't get confused with the words, max runtime and max search result limit has been applied on subsearch so that, subsearch should not take a lot of time to run and block the main search to run.