Getting Data In

Is it possible to configure the memory allocation for a Splunk forwarder to prevent abrupt termination of the splunkd process?

gesman
Communicator

We have a situation where a Splunk forwarder is abruptly dying on one of the servers once a day or so.
Upon further investigation this was discovered:

root@intelsat [/var/log]# cat messages | grep splunkd
Jul 12 06:25:01 intelsat kernel: [30182]     0 30182   706988   480205   1       0             0 splunkd
Jul 12 06:25:01 intelsat kernel: [30183]     0 30183    13200       92   1     -17         -1000 splunkd
Jul 12 06:25:01 intelsat kernel: Out of memory: Kill process 30182 (splunkd) score 260 or sacrifice child
Jul 12 06:25:01 intelsat kernel: Killed process 30182 (splunkd) total-vm:2827952kB, anon-rss:1920820kB, file-rss:0kB
Jul 12 06:25:01 intelsat kernel: splunkd invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0
Jul 12 06:25:01 intelsat kernel: splunkd cpuset=/ mems_allowed=0
Jul 12 06:25:01 intelsat kernel: Pid: 30201, comm: splunkd Not tainted 3.2.13-grsec-xxxx-grs-ipv6-64 #1
Jul 12 06:25:01 intelsat kernel: [30201]     0 30182   706988   480567   3       0             0 splunkd
Jul 12 06:25:01 intelsat kernel: [30183]     0 30183    13200       92   1     -17         -1000 splunkd
Jul 12 06:25:02 intelsat kernel: [30210]     0 30182   706988   480543   3       0             0 splunkd
Jul 12 06:25:02 intelsat kernel: [30183]     0 30183    13200       92   1     -17         -1000 splunkd

As I understand, it's happening because Splunk forwarder requests more memory than available on the system?
Is it possible to configure forwarder for some form of "safe" memory allocation strategy to prevent this from happening?

Ideally i'd want to configure forwarder to auto-restart as well...

0 Karma
1 Solution

MuS
SplunkTrust
SplunkTrust

Hi gesman,

Unfortunately, there is no direct way to limit the memory usage of a forwarder.
But you can achieve it, if you make sure that your inputs are as limited as possible, e.g., point at single files or small directories with a few files and no subdirectories. You can reduce some memory by: shrinking maxQueueSize in outputs.conf to maybe 100 KB (vs a default of 500 KB)

Another thing you can do, is to tell OOM of your OS to be "nicer" to Splunk, even they don't recommend turning it off.
http://www.oracle.com/technetwork/articles/servers-storage-dev/oom-killer-1911807.html

Regarding the auto-restart feature; How should this be handled by Splunk if it is killed by OOM? Create a agent-like wrapper script which checks if splunkd is running and if not, restart Splunk.

Hope this helps ...

cheers, MuS

View solution in original post

MuS
SplunkTrust
SplunkTrust

Hi gesman,

Unfortunately, there is no direct way to limit the memory usage of a forwarder.
But you can achieve it, if you make sure that your inputs are as limited as possible, e.g., point at single files or small directories with a few files and no subdirectories. You can reduce some memory by: shrinking maxQueueSize in outputs.conf to maybe 100 KB (vs a default of 500 KB)

Another thing you can do, is to tell OOM of your OS to be "nicer" to Splunk, even they don't recommend turning it off.
http://www.oracle.com/technetwork/articles/servers-storage-dev/oom-killer-1911807.html

Regarding the auto-restart feature; How should this be handled by Splunk if it is killed by OOM? Create a agent-like wrapper script which checks if splunkd is running and if not, restart Splunk.

Hope this helps ...

cheers, MuS

gesman
Communicator

Thank you.
I think my specific issue is that forwarder actually watching directory tree with 160,000+ files in it.
Although 99% of these files are very rare being modified - not sure of that helps or not.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...