topic Why is there forceful termination of the search process when using stats dc()? in Splunk Search

Why is there forceful termination of the search process when using stats dc()?

the_wolverine — Wed, 31 Jan 2018 18:07:05 GMT

*Forcefully terminated search process with sid=1517416303.2383_ABC123 since its physical memory usage (36521.336000 MB) has exceeded the physical memory threshold specified in limits.conf/search_process_memory_usage_threshold (32768.000000 MB).*

Does anyone have a solution for this issue where using stats dc(field) results in forceful termination of the search? I cannot raise the memory allowance any higher (currently 32Gb) which risks our searcher going down when a user runs this type of query.

Obviously, it is caused by higher distinct counts but it is nothing unreasonable about the query.

Surely, Splunk has seen this many times and has a solution?
Is there some additional configuration that will allow us to workaround the high memory consumption for this type of search?

Re: Why is there forceful termination of the search process when using stats dc()?

davpx — Wed, 31 Jan 2018 23:25:00 GMT

Can you share your search string? How much data is this searching and for what size time window?

Re: Why is there forceful termination of the search process when using stats dc()?

jeanyvesnolen — Thu, 01 Feb 2018 09:41:45 GMT

To reduce DC memory impact you can change

[stats]
dc_digest_bits=9

But the result can be approximative =/

limits.conf reference

Re: Why is there forceful termination of the search process when using stats dc()?

the_wolverine — Fri, 02 Feb 2018 22:17:33 GMT

Yeah, estimate is not ok and in the case where it is estdc can be used. Depending on the data set, sometimes it works and other times still fails due to splunkd forcefully terminated.

Re: Why is there forceful termination of the search process when using stats dc()?

the_wolverine — Tue, 06 Feb 2018 00:50:08 GMT

After dealing with this for a few years, it turns out that when a dc(field) causes out of memory forceful termination, just refactor the query to use :

... | stats count by field | stats count

The tradeoff here is that this type of search will consume more disk, however, a reasonable amount of memory will be consumed which will less likely cause the search to be forcefully terminated due to memory. The new risk now is possible termination caused by exceeding disk quota allocated for search.

Re: Why is there forceful termination of the search process when using stats dc()?

isaiz — Mon, 13 Mar 2023 07:57:26 GMT

I had the same issue, I used that strat and I ran out of search disk memory quota.

What about the chunk_size parameter? Does it make long DC() searches possible?