splunkd is not running. "failed splunkd.pid doesn't exist"

[LMEnterprise]

I have installed Splunk forwarder 9.1.1 on a linux server, but the user and group splunk was unable to be created from the rpm installation. I thought that could have fixed the issue as to why i kept getting an inactive forward-server, but I ended up getting a new error. when i try to restart splunk forwarder, i get the following error:

splunkd is not running. "failed splunkd.pid doesn't exist"

and when i try to have splunk forwarder list the forward-server, I get the following error 3 times:

'tcp_conn_open_afux ossocket_connect failed with no such file or directory'

it still lists my server as an inactive one despite having another splunk forwarder linux host properly connecting to splunk enterprise via ssl connection.

I have also made sure that the listening port (9997) is listened to by splunk. its the same port used by the other linux host to forward logs to

[asimit]

I've seen this exact issue before with Splunk Universal Forwarders. The "splunkd.pid doesn't exist" error combined with the "tcp_conn_open_afux ossocket_connect failed" messages typically happens when there's a conflict between how the Splunk process is started and managed.

Based on your description, this is likely one of two issues:

a. Duplicate systemd service files causing a "split brain" situation
b. Permission problems with the Splunk installation directory

For the first issue, check if you have duplicate service definitions:

ls -la /usr/lib/systemd/system/SplunkForwarder.service
ls -la /etc/systemd/system/SplunkForwarder.service

If both exist, that's causing your problem! The one in /etc/systemd/system takes precedence, and they might have different user/permission settings. You can fix this by:

sudo rm /etc/systemd/system/SplunkForwarder.service
sudo systemctl daemon-reload
sudo systemctl restart SplunkForwarder

If that doesn't work, check the ownership of your Splunk files:

ls -la /opt/splunkforwarder

Make sure everything is owned by the correct user (typically splunk:splunk). If permissions are wrong, you can fix with:

chown -R splunk:splunk /opt/splunkforwarder

As a last resort, the complete reinstall approach works well:

sudo systemctl stop SplunkForwarder
sudo yum remove splunk*
sudo rm -rf /opt/splunkforwarder

Then reinstall the forwarder and configure it properly. I've had good success with this approach when dealing with these mysterious pid and socket connection errors.

Please give 👍 for support 😁 happly splunking .... 😎

[jnoose]

I forgot what it felt like for someone to copy my assignment in homeroom 😂

[jnoose]

I encountered nearly identical symptoms on one of my RHEL9 systems. The key errors you mentioned:

• "splunkd is not running. failed splunkd.pid doesn't exist"
• "tcp_conn_open_afux ossocket_connect failed with no such file or directory"
• Forwarder showing as "inactive" despite correct configuration

  These are classic signs of what I discovered was a "split brain" situation with duplicate systemd service files. While your reinstall fixed it (likely by cleaning up these duplicate files), others might benefit from understanding the root cause:

  Check for duplicate service files:

  ls -la /usr/lib/systemd/system/SplunkForwarder.servicels -la /etc/systemd/system/SplunkForwarder.service

  If both exist, that's the problem! The one in /etc/systemd/system/ takes precedence and might have different user/permissions settings. In my case, one was set to run as SplunkFwd user while the other was running as root.

  This causes a situation where:

  • Systemd shows SplunkForwarder running
  • Splunk CLI thinks it's not running
  • Permission conflicts prevent proper operation
  • PID file issues occur
  • Connection failures happen despite proper network connectivity

    The fix is simpler than reinstalling:

    sudo rm /etc/systemd/system/SplunkForwarder.servicesudo systemctl daemon-reloadsudo systemctl restart SplunkForwarder

    This can happen when multiple installation methods are used (like RPM install + splunk enable boot-start). Sharing this because my "nuke and pave" initially didn't work either until I discovered this specific issue.  Being in the DoD air-gap hell made this even harder to troubleshoot!

[gejimenez]

Hello there, 

I was having the same issue, and it turned out to be a problem with the installation. So, I just did a yum remove splunk* and removed the /opt/splunkforwarder home directory completely.  After, uninstalling and removing the splunk home directory, and I started spunk just fine and I was able to run the add monitor command without any issues. 

I'm running RHEL 8.x and issuing all of these commands via the Linux CLI. Splunk version is 9.1.2. I hope this helps. 

Respectfully. 

Guillermo 

Washington, DC