Installation

Why is Splunk forwarder not starting on linux with error - pid terminated signal 4?

zulfikharnaiyar
Explorer

Hello,

We've installed a linux UF on a supported distribution. Done this on multiple machines but facing an issue on a single server. The forwarder fails to start with the following error:

./splunk start

Starting Splunk...
Splunk> All batbelt. No tights.
Checking prerequisites...
Checking mgmt port [8089]: open
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-8.2.0-e053ef3c985f-linux-2.6-x86_64-manifest'
All installed files intact
Done
All preliminary checks passed.

Starting Splunk server daemon (splunkd)...
ERROR: pid 90702 terminated with signal 4

Done [ OK ]

 

Tried installing it as a daemon as well. It doesn't report any error but the service fails to start. When starting directly from the bin folder, get the above error.

Any helpful pointers here please? 

Thanks.

 

Regards,

Labels (1)
Tags (2)
0 Karma
1 Solution

zulfikharnaiyar
Explorer

Hi @SRG9 ,

Thank you for your input however it was not the case.

The splunkd service was being terminated by a dynatrace agent installed on the machine. Apparently dynatrace agent (liboneagentproc.so) injects  itself to check any service which is being initiated and was terminating the splunk process with an illegal opcode error.

Removing the dynatrace agent solved the issue in our case. However, it would've been ideal if both the solutions could coexist on the same host.

 

View solution in original post

0 Karma

SRG9
Explorer

Hi @zulfikharnaiyar 

I am not sure if below solution works , 

can you rename the splunkd.pid to splunkd.pid_old present in following location and start the splunk?.

/opt/splunkforwarder/var/run/splunk/

0 Karma

zulfikharnaiyar
Explorer

Hi @SRG9 ,

Thank you for your input however it was not the case.

The splunkd service was being terminated by a dynatrace agent installed on the machine. Apparently dynatrace agent (liboneagentproc.so) injects  itself to check any service which is being initiated and was terminating the splunk process with an illegal opcode error.

Removing the dynatrace agent solved the issue in our case. However, it would've been ideal if both the solutions could coexist on the same host.

 

0 Karma

sposani
Engager

I had similar issue, I have applied below config and was able to start Splunk:

$SPLUNK_HOME/etc/system/local/server.conf
[watchdog]
usePreloadedPstacks = false

0 Karma

scelikok
SplunkTrust
SplunkTrust

There is also information on Dynatrace website about Splunk 8.2.

https://www.dynatrace.com/support/help/setup-and-configuration/dynatrace-oneagent/oneaget-troublesho...

There is no workaround yet.

If this reply helps you an upvote and "Accept as Solution" is appreciated.
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Design, Compete, Win: Submit Your Best Splunk Dashboards for a .conf26 Pass

Hello Splunkers,  We’re excited to kick off a Splunk Dashboard contest! We know that dashboards are a primary ...

May 2026 Splunk Expert Sessions: Security & Observability

Level Up Your Operations: May 2026 Splunk Expert Sessions Whether you are refining your security posture or ...

Network to App: Observability Unlocked [May & June Series]

In today’s digital landscape, your environment is no longer confined to the data center. It spans complex ...