Splunk Search

Why is there forceful termination of the search process when using stats dc()?

the_wolverine
Champion
*Forcefully terminated search process with sid=1517416303.2383_ABC123 since its physical memory usage (36521.336000 MB) has exceeded the physical memory threshold specified in limits.conf/search_process_memory_usage_threshold (32768.000000 MB).*

Does anyone have a solution for this issue where using stats dc(field) results in forceful termination of the search? I cannot raise the memory allowance any higher (currently 32Gb) which risks our searcher going down when a user runs this type of query.

Obviously, it is caused by higher distinct counts but it is nothing unreasonable about the query.

Surely, Splunk has seen this many times and has a solution?
Is there some additional configuration that will allow us to workaround the high memory consumption for this type of search?

0 Karma
1 Solution

the_wolverine
Champion

After dealing with this for a few years, it turns out that when a dc(field) causes out of memory forceful termination, just refactor the query to use :

... | stats count by field | stats count

The tradeoff here is that this type of search will consume more disk, however, a reasonable amount of memory will be consumed which will less likely cause the search to be forcefully terminated due to memory. The new risk now is possible termination caused by exceeding disk quota allocated for search.

View solution in original post

0 Karma

the_wolverine
Champion

After dealing with this for a few years, it turns out that when a dc(field) causes out of memory forceful termination, just refactor the query to use :

... | stats count by field | stats count

The tradeoff here is that this type of search will consume more disk, however, a reasonable amount of memory will be consumed which will less likely cause the search to be forcefully terminated due to memory. The new risk now is possible termination caused by exceeding disk quota allocated for search.

0 Karma

jeanyvesnolen
Path Finder

To reduce DC memory impact you can change

[stats]
dc_digest_bits=9

But the result can be approximative =/

limits.conf reference

0 Karma

the_wolverine
Champion

Yeah, estimate is not ok and in the case where it is estdc can be used. Depending on the data set, sometimes it works and other times still fails due to splunkd forcefully terminated.

0 Karma

davpx
Communicator

Can you share your search string? How much data is this searching and for what size time window?

0 Karma
Get Updates on the Splunk Community!

New Cloud Intrusion Detection System Add-on for Splunk

In July 2022 Splunk released the Cloud IDS add-on which expanded Splunk capabilities in security and data ...

Happy CX Day to our Community Superheroes!

Happy 10th Birthday CX Day!What is CX Day? It’s a global celebration recognizing innovation and success in the ...

Check out This Month’s Brand new Splunk Lantern Articles

Splunk Lantern is a customer success center providing advice from Splunk experts on valuable data insights, ...