I got an issue with one of the Universal Forwarder. It is automatically shutting down and when I restart, it is again shutting down immediately. According to what I see when I check status, I figured it out it was the problem with PID files and tried to manually remove them from /opt/splunkforwarder/var/run/splunk . Even after this, I can't find a solution. Can some one help me out?
It was the Ulimits on the box that causes issue. I changed the ulimit values to unlimit after looking into Splunkd logs accordingly and it worked perfectly. May be at first i ignored the WARN message in Splunkd logs as we doesn't pay much attention to warnings 😛