Why is Splunk is restarting itself?

[jason0]

Hello, 

I have a splunk heavy forwarder (splunk 9.0.0.1, centos 7) configured as a heavy forwarder.  When I issue "splunk stop" as either the splunk user or user root, it will restart within a minute.

This system has been configured to start as a systemd service and user splunk is allowed to issue systemctl commands to stop and restart.

just now, I completely disabled the systemd service and then rebooted.  After about a minute of being fully booted, splunk automatically started up.  Its as if there's a parasitic cron or anacron job that starts it if its not running.

What could be causing this?

I would really rather splunk stayed down especially since I am usually altering the configuration files at the time and it starts up before I can complete the task.

--jason

[jason0 · Accepted solution]

Hello,

It's been a few years and I finally had time to figure this out.  There was a SolarWinds agent that was periodically restarting splunk.  I have no idea why, as we had stopped using SolarWinds a while ago.  Uninstalling swiagent fixed it.

To find out what was restarting splunk I ran the following commands (centos 7):

PID=$(pgrep -n splunkd); echo $PID; systemctl status $PID --no-pager
PID=$(pgrep -n splunkd); cat /proc/$PID/cgroup

the output told me splunk was being started by SolarWinds.

Some more information: 

PID=$(pgrep -n splunkd)
ps -o pid,ppid,cmd -p "$PID"
ps -fp "$(ps -o ppid= -p "$PID")"

This will show the parent relationship to the splunk process.

ChatGPT was helpful with the more obscure way to find what was starting splunk.

[PickleRick]

Ha! I knew there was something fishy going on. Splunk shouldn't just restart out of the blue.

Good you found it!

[ashraf_sj]

Any updates on this, have the same situation

[PickleRick]

There are no miracles. If the splunk daemon process is starting "by itself", something must be causing it. Either you have some other form of service management software (monit?) or - much more probable - you have misconfigured your systemd and have multiple units defined for splunk. You can try to find them with

find /etc/systemd -iname \*splunk\*

and/or

find /etc/systemd | xargs grep splunk

[jason0]

First, I can indeed confirm I entered "systemctl disable Splunkd"

Here is the rest:

--------------------------------

systemctl status Splunkd

● Splunkd.service - Systemd service file for Splunk, generated by 'splunk enable boot-start'
Loaded: loaded (/etc/systemd/system/Splunkd.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2023-03-03 13:41:35 PST; 2 days ago
Process: 748 ExecStartPost=/bin/bash -c chown -R splunk:splunk /sys/fs/cgroup/memory/system.slice/%n (code=exited, status=0/SUCCESS)
Process: 745 ExecStartPost=/bin/bash -c chown -R splunk:splunk /sys/fs/cgroup/cpu/system.slice/%n (code=exited, status=0/SUCCESS)
Main PID: 744 (splunkd)
Memory: 1.0G (limit: 7.6G)
CGroup: /system.slice/Splunkd.service
├─ 744 splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_under_systemd
├─ 889 [splunkd pid=744] splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_und... ├─ 1224 /opt/splunk/bin/splunkd instrument-resource-usage -p 8089 --with-kvstore
├─ 1315 /opt/splunk/bin/python3.7 -O /opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrspark... └─23617 /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_secure_gateway/bin/ssg_enable_modular_i...splunk@slsplunksyslog003p:~$ systemctl status Splunkd -l
● Splunkd.service - Systemd service file for Splunk, generated by 'splunk enable boot-start'
Loaded: loaded (/etc/systemd/system/Splunkd.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2023-03-03 13:41:35 PST; 2 days ago
Process: 748 ExecStartPost=/bin/bash -c chown -R splunk:splunk /sys/fs/cgroup/memory/system.slice/%n (code=exited, status=0/SUCCESS)
Process: 745 ExecStartPost=/bin/bash -c chown -R splunk:splunk /sys/fs/cgroup/cpu/system.slice/%n (code=exited, status=0/SUCCESS)
Main PID: 744 (splunkd)
Memory: 1.0G (limit: 7.6G)
CGroup: /system.slice/Splunkd.service
├─ 744 splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_under_systemd
├─ 889 [splunkd pid=744] splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_under_systemd [process-runner
├─ 1224 /opt/splunk/bin/splunkd instrument-resource-usage -p 8089 --with-kvstore
├─ 1315 /opt/splunk/bin/python3.7 -O /opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/root.py --proxied=127.0.0.1,8065,8000
└─23617 /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_secure_gateway/bin/ssg_enable_modular_input.py

----------------------------------------------------------

contents of Splunkd.service:

#This unit file replaces the traditional start-up script for systemd
#configurations, and is used when enabling boot-start for Splunk on
#systemd-based Linux distributions.

[Unit]
Description=Systemd service file for Splunk, generated by 'splunk enable boot-start'
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
Restart=always
ExecStart=/opt/splunk/bin/splunk _internal_launch_under_systemd
KillMode=mixed
KillSignal=SIGINT
TimeoutStopSec=360
LimitNOFILE=65536
LimitNPROC=51200
LimitRTPRIO=99
SuccessExitStatus=51 52
RestartPreventExitStatus=51
RestartForceExitStatus=52
User=splunk
Group=splunk
Delegate=true
CPUShares=1024
MemoryLimit=8201170944
PermissionsStartOnly=true
ExecStartPost=-/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/cpu/system.slice/%n"
ExecStartPost=-/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/memory/system.slice/%n"

[Install]
WantedBy=multi-user.target

[Tom_Lundie]

Could you have an initd script starting Splunk?

ls /etc/init.d/

[jason0]

No.  the only files are:

• /etc/systemd/system/splunkd.service
• /etc/polkit-1/rules.d/10-Splunkd.rules

and there are no SYSV init files that start splunk.

[Tom_Lundie]

Interesting, can you please share the output of:

systemctl status splunkd

 and

cat /etc/systemd/system/splunkd.service

Also just to confirm, to disable splunk you ran:

systemctl disable splunkd