What should the forwarder ulimit -s be?

[ddrillic]

When bringing up a new forwarder, it says -

WARNING: Stack size limit (ulimit -s) is set low (2097152 bytes) Splunk may not work.
You may want to run "ulimit -s unlimited" before starting splunk. 

We see -

[splnkfwd@apsrs3949 dan]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 96030
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 2048
cpu time               (seconds, -t) unlimited
max user processes              (-u) 5120
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

How should we set ulimit and specifically ulimit -s? assuming ulimit -s unlimited is not doable...

[woodcock · Accepted solution]

This setting on the forwarder should be set to accommodate the native function of that server and not to accommodate the splunk forwarder client. The Splunk forwarder client imposes no significant impact to inodes, so as long as your native function is OK, then you should not have to concern yourself with updating this setting when you install the splunk forwarder client. Do you have reason to suspect there is a problem on your forwarder? Of course, if your forwarder exists only for the purpose of being a forwarder and is simply a dumping-ground/way-station for files from other systems to be forwarded into Splunk, then I would say ulimit -s unlimited is the way to go. Why is this not an option?

[ddrillic]

Thank you woodcock! About setting 'ulimit -s unlimited' - we work with all kinds of SAa and it might be an issue or not. The forwarder on this server crashed once already since it was installed just a couple of days ago, so I wonder what the recommendation is in case 'ulimit -s unlimited' is not doable.

[woodcock]

Did it crash because it ran out of inodes? Why would unlimited be a problem? Unless you have a reason to limit inodes, IMHO, you should not do so because miscalculations (shortages) usually cause surprise outages (as you may be experiencing).