Thanks for your reply. I checked the permissions on PID file.
-rw-------. 1 root root 6 Aug 9 05:37 conf-mutator.pid
-rw-r-----. 1 root root 12 Aug 9 05:37 splunkd.pid
Looks like it is accessible only root. Also script to start the splunk is executed with root privileges.
I have a doubt whether the previous version of splunk is not removed cleanly. Will check how to remove it properly.
... View more
I was working on a puppet module to update existing splunkforwarder from 6.1.* to 7.1.7. Installation went smooth but now the forwarder is failing to start with below error message.
systemd: Failed to read PID from file /**/splunk/splunkd.pid: Invalid argument
Centos is the target machine.
Description=Splunk Enterprise 7.1.7
ExecStart=/**/splunk start --accept-license --answer-yes --no-prompt
# If you want to use $(systemctl [start|stop|restart] splunk) instead of splunkd ...
And the puppet script involves below procedure.
Removing files in existing custom installation path of splunkforwarder.
Extracting the 7.1.7 tar ball package to the custom path.
Copying required certs for server connectivity.
Accepting licence and updating the permissions to be accessible by splunk "chown -R splunk:splunk" (This might not be required at all. Since the splunk will be started as root user. But I just thought of having it ).
Moving deploymentclient and web.conf files to the custom path.
For starting the splunk agent.
I just ensured that splunk.service is having necessary permissions
Reload the systemctl daemon and ensure that it is running via puppet code.
I came across couple of posts where the files are not having enough permission. But in my case, since the splunk is started with root privileges I dont see anything like that.
And on few occasions, I was able to successfully start the process (by removing PID file when the process is running) but i dont think its the proper way to start it.
... View more