Getting Data In
Highlighted

Did the forwarder stopped forwarding, possibly due to ulimit?

Engager

Hi Splunkers,

I have the forwarder installed on nix machine. It was working perfectly until today when I made some changes in Inputs.conf to add more log files. When I restarted the forwarder again, it came up and Splunk is restarted successfully but no logs were forwarding.

However, I did face some warning. "Set the Ulimit, Splunk may not work"
Is Ulimit the issue?
If it is, then suddenly why did it stopped working?

Thanks for the help.

0 Karma
Highlighted

Re: Did the forwarder stopped forwarding, possibly due to ulimit?

Explorer

Maybe Splunk is monitoring too many files on your forwarder for the OS to handle. You could try increasing the ulimits:

https://docs.splunk.com/Documentation/Splunk/7.3.0/Troubleshooting/ulimitErrors

I would also ensure you didn't accidentally add a directory with a huge volume of files. I'd double check your inputs.conf.

Highlighted

Re: Did the forwarder stopped forwarding, possibly due to ulimit?

Engager

Thanks ..Can i change the ulimits to unlimited...will it not impact OS performance

0 Karma
Highlighted

Re: Did the forwarder stopped forwarding, possibly due to ulimit?

Explorer

I'd take into account what else the server is doing and how many files you are monitoring, as well as the type of hardware your server is using. Bumping up the ulimits will allow the OS to monitor more files but at a cost of performance.

0 Karma
Highlighted

Re: Did the forwarder stopped forwarding, possibly due to ulimit?

Builder

The below post has helped many on this issue. You'd want to check this -
https://answers.splunk.com/answers/13313/how-to-tune-ulimit-on-my-server.html

0 Karma
Highlighted

Re: Did the forwarder stopped forwarding, possibly due to ulimit?

Engager

Hi.So ulimit may be the reason it suddenly stopped forwarding when i restarted ??
It was working fine previously

0 Karma
Highlighted

Re: Did the forwarder stopped forwarding, possibly due to ulimit?

Builder

yes, this can be the reason as your errors are directly pointing to that. Can you make sure, the additional monitoring that you added, how many files and what size are they ? you can check for the resources usage by splunkd on you m/c to see for the performance.
If the additional monitoring requires splunk to open too many file descriptors but the defined ulimit is not sufficient, you'd face this problem

0 Karma
Highlighted

Re: Did the forwarder stopped forwarding, possibly due to ulimit?

Ultra Champion

On the forwarder, for the proper id, what does the ulimit command show - ulimit -n?

0 Karma
Highlighted

Re: Did the forwarder stopped forwarding, possibly due to ulimit?

Engager

This is the present setting

time(seconds) unlimited
file(blocks) 2097151
data(kbytes) unlimited
stack(kbytes) 32768
memory(kbytes) unlimited
coredump(blocks) 2097151
nofiles(descriptors) 2000
threads(per process) unlimited
processes(per user) unlimited

0 Karma
Highlighted

Re: Did the forwarder stopped forwarding, possibly due to ulimit?

Ultra Champion

nofiles(descriptors) 2000 is almost the minimum - it should be higher.

0 Karma