Why is the percentage of non high priority searches skipped (50%) is high?

[dejiosemeke]

The percentage of non high priority searches skipped (50%) over the last 24 hours is very high and exceeded the red thresholds (20%) on this Splunk instance. Total Searches that were part of this percentage=2. Total skipped Searches=1

[richgalloway]

What is your question?

[timpacl]

I see this error reported by indexers. That seems wrong since a "skipped" search is never dispatched to an indexer. Am I misunderstanding the message? "The percentage of non high priority searches skipped (33%) over the last 24 hours is very high and exceeded the red thresholds (20%) on this Splunk instance."

[PickleRick]

Not really. There are two separate limits that you have to take into accounts when dealing with searches.

One is the limit of concurrent searches at search-head. And another one is the limit at indexer.

So if you have several search-heads (clustered or not), they can have their limits set high enough that the combined searches from several search-heads exceed the limit set for the indexer. The searches then can be delayed or skipped.

[timpacl]

Thanks for the response. Interesting. Splunk PS told us that indexers don't have a search limit. Do you know when the search limit was implemented for indexers? Where it is configured?

[PickleRick]

Of course in limits.conf 🙂

https://docs.splunk.com/Documentation/Splunk/8.2.5/Admin/Limitsconf#Concurrency

As far as I understand, those limits apply both to search heads and indexers.

You have to remember that indexer uses some part of its capacity for indexing process (typically 4-6CPUs per indexing pipeline if I remember correctly) so you have less CPUs available for searching.

[timpacl]

Thanks. Running btool on an indexer returns settings that aren't mentioned in the docs and none of the settings that are mentioned in the docs:

Documentation

Concurrency

base_max_searches = <integer>
max_rt_search_multiplier = <decimal number>
max_searches_per_cpu = <integer>

Indexer "splunk btool limits list concurrency --debug" (all setting returned from 😞
/opt/splunk/etc/system/default/limits.conf [concurrency]
/opt/splunk/etc/system/default/limits.conf  max_count = 10000000
/opt/splunk/etc/system/default/limits.conf  max_mem_usage_mb = 200

So, if this is right, my indexers would be trying to run in excess of 10M searches concurrently. We have been tracking daily searches and we are doing 2M/day. Even if subsearches and the like are factored in, I don't see how we could hit 10M concurrent searches.

Thanks for helping with this. It has provided direction to look.

[PickleRick]

You're checking in the wrong section.

The [concurrency] stanza is for something different. The settings we're talking about are indeed concurrency settings but within the [search] stanza.

And they are perfectly well defined in system/default/limits.conf.

[timpacl]

If the search stanza settings are being applied then each indexer would be limited to 30 concurrent searches with the default settings (which is what we have). We asked Splunk Professional Services about that and they said these limits weren't enforced on the indexers. Rough math says that if every search ran was just 1 second duration a would be limited to 2.5M searches/day. As I said, we are doing 2M/day now and our searches aren't all 1 sec. Last year we had a peak at 3.25M searches/day.

[PickleRick]

No, max_count is a different story.

You want these parameters:

base_max_searches = <integer>
* A constant to add to the maximum number of searches, computed as a
  multiplier of the CPUs.
* Default: 6

max_rt_search_multiplier = <decimal number>
* A number by which the maximum number of historical searches is multiplied
  to determine the maximum number of concurrent real-time searches.
* NOTE: The maximum number of real-time searches is computed as:
  max_rt_searches = max_rt_search_multiplier x max_hist_searches
* Default: 1

max_searches_per_cpu = <integer>
* The maximum number of concurrent historical searches for each CPU.
  The system-wide limit of historical searches is computed as:
  max_hist_searches =  max_searches_per_cpu x number_of_cpus + base_max_searches
* NOTE: The maximum number of real-time searches is computed as:
  max_rt_searches = max_rt_search_multiplier x max_hist_searches
* Default: 1

[timpacl]

I understand how those settings work, but they aren't set on the indexers. Seems like there should be no limit to search unless max_count is a limit, which you are saying it isn't. Is there a hard-coded limit in the Splunk code somewhere?

[PickleRick]

OK. It seems strange. On the one hand, it does indeed seem to not be enforced - I lowered it to 0 (zero!) per cpu and just 1 base search. Which should give me just one available search per indexer.

I'm still able to run three parallel real-time searches against my indexers even though my monitoring console says that the limit is 1 search. But the utilization is 3/1 🙂

So it might indeed not be enforced. Sorry for confusion then.