I have the forwarder installed on nix machine. It was working perfectly until today when I made some changes in Inputs.conf to add more log files. When I restarted the forwarder again, it came up and Splunk is restarted successfully but no logs were forwarding.
However, I did face some warning.
"Set the Ulimit, Splunk may not work"
Is Ulimit the issue?
If it is, then suddenly why did it stopped working?
Thanks for the help.
Maybe Splunk is monitoring too many files on your forwarder for the OS to handle. You could try increasing the ulimits:
I would also ensure you didn't accidentally add a directory with a huge volume of files. I'd double check your inputs.conf.
I'd take into account what else the server is doing and how many files you are monitoring, as well as the type of hardware your server is using. Bumping up the ulimits will allow the OS to monitor more files but at a cost of performance.
yes, this can be the reason as your errors are directly pointing to that. Can you make sure, the additional monitoring that you added, how many files and what size are they ? you can check for the resources usage by splunkd on you m/c to see for the performance.
If the additional monitoring requires splunk to open too many file descriptors but the defined ulimit is not sufficient, you'd face this problem
This is the present setting
threads(per process) unlimited
processes(per user) unlimited