I was working on a puppet module to update existing splunkforwarder from 6.1.* to 7.1.7. Installation went smooth but now the forwarder is failing to start with below error message.
systemd: Failed to read PID from file /**/splunk/splunkd.pid: Invalid argument
Centos is the target machine.
Description=Splunk Enterprise 7.1.7
ExecStart=/**/splunk start --accept-license --answer-yes --no-prompt
# If you want to use $(systemctl [start|stop|restart] splunk) instead of splunkd ...
And the puppet script involves below procedure.
Removing files in existing custom installation path of splunkforwarder.
Extracting the 7.1.7 tar ball package to the custom path.
Copying required certs for server connectivity.
Accepting licence and updating the permissions to be accessible by splunk "chown -R splunk:splunk" (This might not be required at all. Since the splunk will be started as root user. But I just thought of having it ).
Moving deploymentclient and web.conf files to the custom path.
For starting the splunk agent.
I just ensured that splunk.service is having necessary permissions
Reload the systemctl daemon and ensure that it is running via puppet code.
I came across couple of posts where the files are not having enough permission. But in my case, since the splunk is started with root privileges I dont see anything like that.
And on few occasions, I was able to successfully start the process (by removing PID file when the process is running) but i dont think its the proper way to start it.
I have seen PID issues when Splunk Enterprise or Splunk Forwarders have been started as root, but are now being started as another user. The PID file is owned by root and won't let the other user start it. Check your permissions on the PID file and verify the user you are starting Splunk as.
It is also best practice to run the forwarders as a non-root user.