Installation

Indexer crashing repeatedly

khusain_splunk
Splunk Employee
Splunk Employee

We did a recent upgrade to 7.0.9 on our environment and started Splunk on the indexer. But again, the moment we started it, takes roughly around 20 min - 1 hour and then Splunk crashes on this. Same old story

Last FATAL error is as follows

03-29-2019 05:22:51.000 -0400 FATAL ProcessRunner - Unexpected EOF from process runner child!
03-29-2019 05:22:51.000 -0400 ERROR ProcessRunner - helper process seems to have died (child killed by signal 15: Terminated)!

Tags (1)
0 Karma

anaidu_splunk
Splunk Employee
Splunk Employee

TO remediate this you can follow the below steps which also worked for me.

  1. Enabled the boot-start with splunk user.

  2. Check the directory permission which is set to splunk:splunk

  3. Exit the login session and you will find that Plunk again stopped.

  4. This means that enable boot-start has not created systemd service.

  5. Now manually create the splunkd.service under systemd/system. Unit file used is below

  6. Set the parameter "RemainAfterExit=yes" and reloaded the daemon.

  7. Restarted the Splunkd with systemctl.

  8. Exit the login session again and observer that Splunk service which should not be stopped ans should resolve the issue.

Navigate to:- /etc/systemd/system/splunkd.service
[Unit]
Description=Splunk Enterprise 6.5.0
After=network.target
Wants=network.target

[Service]
Type=forking
RemainAfterExit=False
User=splunk
Group=splunk
LimitNOFILE=65536
ExecStart=/opt/splunk/bin/splunk start --accept-license --answer-yes --no-prompt
ExecStop=/opt/splunk/bin/splunk stop
PIDFile=/opt/splunk/var/run/splunk/splunkd.pid

[Install]
WantedBy=multi-user.target

If you want to use $(systemctl [start|stop|restart] splunk) instead of splunkd ...

Alias=splunk.service

If you still see the Splunk is crashing with the below message from /var/log/messages use the below parameter:

May 18 21:30:56 pplsplunkapph25 abrt-server[6936]: Executable '/opt/splunk/bin/splunkd' doesn't belong to any package and ProcessUnpackaged is set to 'no'

To solve this limitation, you need to edit the file

sudo vim /etc/abrt/abrt-action-save-package-data.conf
Then change the parameter ProcessUnpackaged to yes

Process crashes in executables which do not belong to any package?

ProcessUnpackaged = yes

If still Splunk crashes, please raise a ticket with your internal admin team.

REF DOC:-
https://docs.splunk.com/Documentation/Splunk/7.2.6/Admin/RunSplunkassystemdservice

https://answers.splunk.com/answers/59662/is-there-a-systemd-unit-file-for-splunk.html

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...