Splunk Enterprise

How to resolve this error: splunk service restart due to Checking max_mem_usage_mb resultsSize?

jinnypt
Explorer

Hello. 

The splunk service is restarting with an error as shown below during report scheduling execution at a specific time period.

 

search fail log

08-17-2022 06:00:04.164 INFO SearchOperator:inputcsv [12673 phase_1] - sid:scheduler__admin__search__RMD52e8470291689a839_at_1660683600_5272 Successfully read lookup file '/opt/splunk/etc/apps/search/lookups/xxx.csv'.
08-17-2022 06:00:04.166 INFO MultiValueProcessor [12673 phase_1] - Checking max_mem_usage_mb resultsSize=100 maxHeapSize=15728640000 memoryUsage=1824925 earlyExit=0
08-17-2022 06:00:04.169 INFO MultiValueProcessor [12673 phase_1] - Checking max_mem_usage_mb resultsSize=200 maxHeapSize=15728640000 memoryUsage=6273048 earlyExit=0
08-17-2022 06:00:04.170 INFO MultiValueProcessor [12673 phase_1] - Checking max_mem_usage_mb resultsSize=300 maxHeapSize=15728640000 memoryUsage=7531940 earlyExit=0
....
08-17-2022 06:00:06.484 INFO MultiValueProcessor [12673 phase_1] - Checking max_mem_usage_mb resultsSize=25200 maxHeapSize=15728640000 memoryUsage=531030711 earlyExit=0
08-17-2022 06:00:06.485 INFO MultiValueProcessor [12673 phase_1] - Checking max_mem_usage_mb resultsSize=25300 maxHeapSize=15728640000 memoryUsage=531809607 earlyExit=0
08-17-2022 06:00:13.237 FATAL ProcessRunner [9783 ProcessRunner] - Unexpected EOF from process runner child!
08-17-2022 06:00:13.238 FATAL ProcessRunner [9783 ProcessRunner] - Helper process was killed by SIGKILL. Usually this indicates that the kernel's OOM-killer has decided to terminate the daemon process.
08-17-2022 06:00:13.238 FATAL ProcessRunner [9783 ProcessRunner] - Check the kernel log (possibly /var/log/messages) for more info
08-17-2022 06:00:13.238 ERROR ProcessRunner [9783 ProcessRunner] - helper process seems to have died (child killed by signal 9: Killed)!

 

Splunk config information

/opt/splunk/etc/system/local/limit.conf

[default]
max_mem_usage_mb = 30000

 

/opt/splunk/etc/apps/search/local/limit.conf

[default]
max_mem_usage_mb = 10000

 

Even with the above settings, it seems that the memory is not actually used as much as the settings.

Splunk Spec: 16core, 64GB

 

If anyone knows about this issue, please share.

Labels (1)
0 Karma

haph
Path Finder

Hi, any solutions for this in the meantime?

I'm experiencing the same issue under 8.3.3.

 

Thanks!

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

based on log events, I assume that there was some lookup which contains some multivalue fields. Officially those are supported only on kv store lookups not on CSVs.

I propose that you create a support case to Splunk Support to get more information about this.

r. Ismo

Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...