topic Re: Terminated with signal 4 (core dumped) when upgrading in Splunk Enterprise

Terminated with signal 4 (core dumped) when upgrading

leahs — Mon, 02 Aug 2021 13:57:08 GMT

Hi,

I'm upgrading my cluster master from version 8.0.3 to 8.2.1. After installing the new version over the old deployment and starting splunk, I get "ERROR: pid xxx terminated with signal 4 (core dumped)", and the Splunk web server is not available. How can I fix this?

My Splunk environment is running on AWS Linux EC2s. This is the information i have about the OS:

NAME="Amazon Linux AMI"
VERSION="2018.03"
ID_LIKE="rhel fedora"

Re: Terminated with signal 4 (core dumped) when upgrading

jonoped — Mon, 02 Aug 2021 05:05:34 GMT

Hi @leahs ,

Did you manage to find a solution to this?
I am receiving the same error, though mine is a fresh build of an on-prem VM not cloud.

Thanks,

Jono

Re: Terminated with signal 4 (core dumped) when upgrading

leahs — Mon, 02 Aug 2021 13:56:06 GMT

Hi @jonoped,

I haven't solved the problem yet. The instances are running on this OS:

NAME="Amazon Linux AMI"
VERSION="2018.03"
ID_LIKE="rhel fedora"

What OS are you running?

Lea

Re: Terminated with signal 4 (core dumped) when upgrading

anirbandasdeb — Mon, 13 Dec 2021 20:09:29 GMT

Were you able to get any resolution for this @leahs ?

Im facing the exact same issue while doing a dry run in preparation for the actual upgrade..

Checking prerequisites... Checking http port [9443]: open Checking mgmt port [8089]: open Checking appserver port [127.0.0.1:8065]: open Checking kvstore port [8191]: open Checking configuration... Done. Checking critical directories... Done Checking indexes... Validated: _audit _internal _introspection _metrics _metrics_rollup _telemetry _thefishbucket history main summary Done Checking filesystem compatibility... Done Checking conf files for problems... Done Checking default conf files for edits... Validating installed files against hashes from '/opt/splunk/splunk-8.2.1-ddff1c41e5cf-linux-2.6-x86_64-manifest' All installed files intact. Done Checking replication_port port [23456]: open All preliminary checks passed. Starting splunk server daemon (splunkd)... ERROR: pid 2283 terminated with signal 4 (core dumped) Done [ OK ] Waiting for web server at http://127.0.0.1:9443 to be available..............................................................................................................................................................................................................^C (dev2) splunk@splnkhfvm-xxx:~ $ ./bin/splunk version Splunk 8.2.1 (build ddff1c41e5cf)

I am upgrading an SH Cluster from v8.0.5 to 8.2.1

(dev2) splunk@splnkhfvm-xxx:~ $ hostnamectl Static hostname: splnkhfvm-qvjj5.xxx Icon name: computer-vm Chassis: vm Machine ID: xxx Boot ID: xxx Virtualization: kvm Operating System: Red Hat Enterprise Linux CPE OS Name: cpe:/o:redhat:enterprise_linux:7.9:GA:server Kernel: Linux 3.10.0-1160.15.2.el7.x86_64 Architecture: x86-64 (dev2) splunk@splnkhfvm-xxx:~ $

Entry in /var/log/messages log file

Aug 3 18:04:45 splnkhfvm-xxx sudo: anirban : TTY=pts/1 ; PWD=/home/dasd ; USER=splunk ; COMMAND=/opt/splunk/bin/splunk start Aug 3 18:04:49 splnkhfvm-qvjj5 kernel: [7308988.389979] traps: splunkd[5860] trap invalid opcode ip:7f6a374da3bf sp:7ffda55435b0 error:0 in liboneagentproc.so[7f6a374c6000+84000]

Re: Terminated with signal 4 (core dumped) when upgrading

jasen_m — Mon, 09 Aug 2021 19:05:42 GMT

Hello, we encountered this same issue when we also tried to upgrade and I actually saw your post when we were looking for a solution.

Ultimately, we were able to find out that there was a conflict with something else we had installed on the host.

Can you try to look in /var/log/messages after trying to start splunk?

In our case, we observed the following message:
Aug 7 00:22:54 <splunk_host> kernel: splunkd[53109] trap invalid opcode ip:XXXXXXX sp:XXXXXXXXXX error:0 in <non_splunk_agent>.so[XXXXXX+XXXXX]

After we saw that, we were able to uninstall the agent and splunk was able to start normally on the search head.

Re: Terminated with signal 4 (core dumped) when upgrading

leahs — Tue, 10 Aug 2021 08:13:46 GMT

Hi @jasen_m ,

Thank you so much!! This solved the issue.

Re: Terminated with signal 4 (core dumped) when upgrading

pignardh — Thu, 07 Oct 2021 13:45:13 GMT

Hello,
I had the same problem and found this solution:

You must deactivate the "usePreloadedPstacks" parameter in
$SPLUNK_HOME/etc/system/local/servers.conf
[watchdog]
usePreloadedPstacks = false

Re: Terminated with signal 4 (core dumped) when upgrading

vzabawski — Mon, 08 Nov 2021 11:01:30 GMT

To whoever might find this interesting.

I've recently encountered with such issue after installing Dynatrace OneAgent chart in the same k8s cluster with Splunk. In my case I wasn't able to just delete the liboneagentproc.so file, so I had to uninstall Dynatrace chart and then delete /opt/oneagent directory in pod where Splunk runs.

Link to upgrade notes: https://docs.splunk.com/Documentation/Splunk/8.2.1/ReleaseNotes/Knownissues