Hi Splunkers,
I have the forwarder installed on nix machine. It was working perfectly until today when I made some changes in Inputs.conf to add more log files. When I restarted the forwarder again, it came up and Splunk is restarted successfully but no logs were forwarding.
However, I did face some warning. "Set the Ulimit, Splunk may not work"
Is Ulimit the issue?
If it is, then suddenly why did it stopped working?
Thanks for the help.
Yes, this absolutely could, and based on the warning, probably has caused the problem. You have run out of address space to store open file descriptors for splunk:
http://www.georgestarcher.com/splunk-ulimits-and-you/
On the forwarder, for the proper id, what does the ulimit command show - ulimit -n
?
This is the present setting
time(seconds) unlimited
file(blocks) 2097151
data(kbytes) unlimited
stack(kbytes) 32768
memory(kbytes) unlimited
coredump(blocks) 2097151
nofiles(descriptors) 2000
threads(per process) unlimited
processes(per user) unlimited
nofiles(descriptors) 2000
is almost the minimum - it should be higher.
The below post has helped many on this issue. You'd want to check this -
https://answers.splunk.com/answers/13313/how-to-tune-ulimit-on-my-server.html
Hi.So ulimit may be the reason it suddenly stopped forwarding when i restarted ??
It was working fine previously
yes, this can be the reason as your errors are directly pointing to that. Can you make sure, the additional monitoring that you added, how many files and what size are they ? you can check for the resources usage by splunkd on you m/c to see for the performance.
If the additional monitoring requires splunk to open too many file descriptors but the defined ulimit is not sufficient, you'd face this problem
Maybe Splunk is monitoring too many files on your forwarder for the OS to handle. You could try increasing the ulimits:
https://docs.splunk.com/Documentation/Splunk/7.3.0/Troubleshooting/ulimitErrors
I would also ensure you didn't accidentally add a directory with a huge volume of files. I'd double check your inputs.conf.
Thanks ..Can i change the ulimits to unlimited...will it not impact OS performance
I'd take into account what else the server is doing and how many files you are monitoring, as well as the type of hardware your server is using. Bumping up the ulimits will allow the OS to monitor more files but at a cost of performance.