Is it possible to configure the memory allocation for a Splunk forwarder to prevent abrupt termination of the splunkd process?

[gesman]

We have a situation where a Splunk forwarder is abruptly dying on one of the servers once a day or so.
Upon further investigation this was discovered:

root@intelsat [/var/log]# cat messages | grep splunkd
Jul 12 06:25:01 intelsat kernel: [30182]     0 30182   706988   480205   1       0             0 splunkd
Jul 12 06:25:01 intelsat kernel: [30183]     0 30183    13200       92   1     -17         -1000 splunkd
Jul 12 06:25:01 intelsat kernel: Out of memory: Kill process 30182 (splunkd) score 260 or sacrifice child
Jul 12 06:25:01 intelsat kernel: Killed process 30182 (splunkd) total-vm:2827952kB, anon-rss:1920820kB, file-rss:0kB
Jul 12 06:25:01 intelsat kernel: splunkd invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0
Jul 12 06:25:01 intelsat kernel: splunkd cpuset=/ mems_allowed=0
Jul 12 06:25:01 intelsat kernel: Pid: 30201, comm: splunkd Not tainted 3.2.13-grsec-xxxx-grs-ipv6-64 #1
Jul 12 06:25:01 intelsat kernel: [30201]     0 30182   706988   480567   3       0             0 splunkd
Jul 12 06:25:01 intelsat kernel: [30183]     0 30183    13200       92   1     -17         -1000 splunkd
Jul 12 06:25:02 intelsat kernel: [30210]     0 30182   706988   480543   3       0             0 splunkd
Jul 12 06:25:02 intelsat kernel: [30183]     0 30183    13200       92   1     -17         -1000 splunkd

As I understand, it's happening because Splunk forwarder requests more memory than available on the system?
Is it possible to configure forwarder for some form of "safe" memory allocation strategy to prevent this from happening?

Ideally i'd want to configure forwarder to auto-restart as well...

[MuS · Accepted solution]

Hi gesman,

Unfortunately, there is no direct way to limit the memory usage of a forwarder.
But you can achieve it, if you make sure that your inputs are as limited as possible, e.g., point at single files or small directories with a few files and no subdirectories. You can reduce some memory by: shrinking maxQueueSize in outputs.conf to maybe 100 KB (vs a default of 500 KB)

Another thing you can do, is to tell OOM of your OS to be "nicer" to Splunk, even they don't recommend turning it off.
http://www.oracle.com/technetwork/articles/servers-storage-dev/oom-killer-1911807.html

Regarding the auto-restart feature; How should this be handled by Splunk if it is killed by OOM? Create a agent-like wrapper script which checks if splunkd is running and if not, restart Splunk.

Hope this helps ...

cheers, MuS

[gesman]

Thank you.
I think my specific issue is that forwarder actually watching directory tree with 160,000+ files in it.
Although 99% of these files are very rare being modified - not sure of that helps or not.