I was working on a puppet module to update existing splunkforwarder from 6.1.* to 7.1.7. Installation went smooth but now the forwarder is failing to start with below error message.
systemd: Failed to read PID from file /**/splunk/splunkd.pid: Invalid argument
Centos is the target machine.
[Unit] Description=Splunk Enterprise 7.1.7 After=network.target Wants=network.target [Service] Type=forking RemainAfterExit=False User=root Group=root LimitNOFILE=65536 ExecStart=/**/splunk start --accept-license --answer-yes --no-prompt ExecStop=/**/splunkstop PIDFile=/**/splunkforwarder/var/run/splunk/splunkd.pid Restart=always [Install] WantedBy=multi-user.target # If you want to use $(systemctl [start|stop|restart] splunk) instead of splunkd ... Alias=gsosplunk.service
And the puppet script involves below procedure.
For starting the splunk agent.
I just ensured that splunk.service is having necessary permissions
Reload the systemctl daemon and ensure that it is running via puppet code.
I came across couple of posts where the files are not having enough permission. But in my case, since the splunk is started with root privileges I dont see anything like that.
And on few occasions, I was able to successfully start the process (by removing PID file when the process is running) but i dont think its the proper way to start it.
I have seen PID issues when Splunk Enterprise or Splunk Forwarders have been started as root, but are now being started as another user. The PID file is owned by root and won't let the other user start it. Check your permissions on the PID file and verify the user you are starting Splunk as.
It is also best practice to run the forwarders as a non-root user.
Thanks for your reply. I checked the permissions on PID file.
-rw-------. 1 root root 6 Aug 9 05:37 conf-mutator.pid -rw-r-----. 1 root root 12 Aug 9 05:37 splunkd.pid
Looks like it is accessible only root. Also script to start the splunk is executed with root privileges.
I have a doubt whether the previous version of splunk is not removed cleanly. Will check how to remove it properly.