Indexer crashing repeatedly

[khusain_splunk]

We did a recent upgrade to 7.0.9 on our environment and started Splunk on the indexer. But again, the moment we started it, takes roughly around 20 min - 1 hour and then Splunk crashes on this. Same old story

Last FATAL error is as follows

03-29-2019 05:22:51.000 -0400 FATAL ProcessRunner - Unexpected EOF from process runner child!
03-29-2019 05:22:51.000 -0400 ERROR ProcessRunner - helper process seems to have died (child killed by signal 15: Terminated)!

[anaidu_splunk]

TO remediate this you can follow the below steps which also worked for me.

1. Enabled the boot-start with splunk user.

2. Check the directory permission which is set to splunk:splunk

3. Exit the login session and you will find that Plunk again stopped.

4. This means that enable boot-start has not created systemd service.

5. Now manually create the splunkd.service under systemd/system. Unit file used is below

6. Set the parameter "RemainAfterExit=yes" and reloaded the daemon.

7. Restarted the Splunkd with systemctl.

8. Exit the login session again and observer that Splunk service which should not be stopped ans should resolve the issue.

Navigate to:- /etc/systemd/system/splunkd.service
[Unit]
Description=Splunk Enterprise 6.5.0
After=network.target
Wants=network.target

[Service]
Type=forking
RemainAfterExit=False
User=splunk
Group=splunk
LimitNOFILE=65536
ExecStart=/opt/splunk/bin/splunk start --accept-license --answer-yes --no-prompt
ExecStop=/opt/splunk/bin/splunk stop
PIDFile=/opt/splunk/var/run/splunk/splunkd.pid

[Install]
WantedBy=multi-user.target

If you want to use $(systemctl [start|stop|restart] splunk) instead of splunkd ...

Alias=splunk.service

If you still see the Splunk is crashing with the below message from /var/log/messages use the below parameter:

May 18 21:30:56 pplsplunkapph25 abrt-server[6936]: Executable '/opt/splunk/bin/splunkd' doesn't belong to any package and ProcessUnpackaged is set to 'no'

To solve this limitation, you need to edit the file

sudo vim /etc/abrt/abrt-action-save-package-data.conf
Then change the parameter ProcessUnpackaged to yes

Process crashes in executables which do not belong to any package?

ProcessUnpackaged = yes

If still Splunk crashes, please raise a ticket with your internal admin team.

REF DOC:-
https://docs.splunk.com/Documentation/Splunk/7.2.6/Admin/RunSplunkassystemdservice

https://answers.splunk.com/answers/59662/is-there-a-systemd-unit-file-for-splunk.html