Splunk Enterprise Forwarder -7.1.7 - Failed to Start - Error "Failed to read PID from file Invalid Argument"

[nandhak21]

Hi,

I was working on a puppet module to update existing splunkforwarder from 6.1.* to 7.1.7. Installation went smooth but now the forwarder is failing to start with below error message.

systemd[1]: Failed to read PID from file /**/splunk/splunkd.pid: Invalid argument

Centos is the target machine.

splunk.service file

[Unit]
Description=Splunk Enterprise 7.1.7
After=network.target
Wants=network.target

[Service]
Type=forking
RemainAfterExit=False
User=root
Group=root
LimitNOFILE=65536
ExecStart=/**/splunk start --accept-license --answer-yes --no-prompt
ExecStop=/**/splunkstop
PIDFile=/**/splunkforwarder/var/run/splunk/splunkd.pid
Restart=always

[Install]
WantedBy=multi-user.target
# If you want to use $(systemctl [start|stop|restart] splunk) instead of splunkd ...
Alias=gsosplunk.service

And the puppet script involves below procedure.

1. Removing files in existing custom installation path of splunkforwarder.
2. Extracting the 7.1.7 tar ball package to the custom path.
3. Copying required certs for server connectivity.
4. Accepting licence and updating the permissions to be accessible by splunk "chown -R splunk:splunk" (This might not be required at all. Since the splunk will be started as root user. But I just thought of having it ).
5. Moving deploymentclient and web.conf files to the custom path.

For starting the splunk agent.

I just ensured that splunk.service is having necessary permissions
Reload the systemctl daemon and ensure that it is running via puppet code.

I came across couple of posts where the files are not having enough permission. But in my case, since the splunk is started with root privileges I dont see anything like that.

And on few occasions, I was able to successfully start the process (by removing PID file when the process is running) but i dont think its the proper way to start it.

[kmorris_splunk]

I have seen PID issues when Splunk Enterprise or Splunk Forwarders have been started as root, but are now being started as another user. The PID file is owned by root and won't let the other user start it. Check your permissions on the PID file and verify the user you are starting Splunk as.

It is also best practice to run the forwarders as a non-root user.

[nandhak21]

Thanks for your reply. I checked the permissions on PID file.

-rw-------. 1 root   root       6 Aug  9 05:37 conf-mutator.pid
-rw-r-----. 1 root   root      12 Aug  9 05:37 splunkd.pid

Looks like it is accessible only root. Also script to start the splunk is executed with root privileges.

I have a doubt whether the previous version of splunk is not removed cleanly. Will check how to remove it properly.

[sloshburch]

For reference: Run Splunk Enterprise as a different or non-root user. Applies to forwarders as well.

[sloshburch]

Also, it sounds like you are removing the old installation and not uninstalling it properly.

Even better would be to upgrade the forwarder in place and not uninstall first.