topic Re: Why is Splunk is restarting itself? in Splunk Enterprise

Why is Splunk is restarting itself?

jason0 — Mon, 06 Mar 2023 14:26:04 GMT

Hello,

I have a splunk heavy forwarder (splunk 9.0.0.1, centos 7) configured as a heavy forwarder. When I issue "splunk stop" as either the splunk user or user root, it will restart within a minute.

This system has been configured to start as a systemd service and user splunk is allowed to issue systemctl commands to stop and restart.

just now, I completely disabled the systemd service and then rebooted. After about a minute of being fully booted, splunk automatically started up. Its as if there's a parasitic cron or anacron job that starts it if its not running.

What could be causing this?

I would really rather splunk stayed down especially since I am usually altering the configuration files at the time and it starts up before I can complete the task.

--jason

Re: splunk is restarting itself.

Tom_Lundie — Sun, 05 Mar 2023 17:21:51 GMT

Could you have an initd script starting Splunk?

ls /etc/init.d/

Re: splunk is restarting itself.

jason0 — Mon, 06 Mar 2023 17:38:18 GMT

No. the only files are:

/etc/systemd/system/splunkd.service
/etc/polkit-1/rules.d/10-Splunkd.rules

and there are no SYSV init files that start splunk.

Re: splunk is restarting itself.

Tom_Lundie — Mon, 06 Mar 2023 17:59:40 GMT

Interesting, can you please share the output of:

systemctl status splunkd

and

cat /etc/systemd/system/splunkd.service

Also just to confirm, to disable splunk you ran:

systemctl disable splunkd

Re: Why is Splunk is restarting itself?

jason0 — Mon, 06 Mar 2023 18:11:07 GMT

First, I can indeed confirm I entered "systemctl disable Splunkd"

Here is the rest:

--------------------------------

systemctl status Splunkd

● Splunkd.service - Systemd service file for Splunk, generated by 'splunk enable boot-start'
Loaded: loaded (/etc/systemd/system/Splunkd.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2023-03-03 13:41:35 PST; 2 days ago
Process: 748 ExecStartPost=/bin/bash -c chown -R splunk:splunk /sys/fs/cgroup/memory/system.slice/%n (code=exited, status=0/SUCCESS)
Process: 745 ExecStartPost=/bin/bash -c chown -R splunk:splunk /sys/fs/cgroup/cpu/system.slice/%n (code=exited, status=0/SUCCESS)
Main PID: 744 (splunkd)
Memory: 1.0G (limit: 7.6G)
CGroup: /system.slice/Splunkd.service
├─ 744 splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_under_systemd
├─ 889 [splunkd pid=744] splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_und... ├─ 1224 /opt/splunk/bin/splunkd instrument-resource-usage -p 8089 --with-kvstore
├─ 1315 /opt/splunk/bin/python3.7 -O /opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrspark... └─23617 /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_secure_gateway/bin/ssg_enable_modular_i...splunk@slsplunksyslog003p:~$ systemctl status Splunkd -l
● Splunkd.service - Systemd service file for Splunk, generated by 'splunk enable boot-start'
Loaded: loaded (/etc/systemd/system/Splunkd.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2023-03-03 13:41:35 PST; 2 days ago
Process: 748 ExecStartPost=/bin/bash -c chown -R splunk:splunk /sys/fs/cgroup/memory/system.slice/%n (code=exited, status=0/SUCCESS)
Process: 745 ExecStartPost=/bin/bash -c chown -R splunk:splunk /sys/fs/cgroup/cpu/system.slice/%n (code=exited, status=0/SUCCESS)
Main PID: 744 (splunkd)
Memory: 1.0G (limit: 7.6G)
CGroup: /system.slice/Splunkd.service
├─ 744 splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_under_systemd
├─ 889 [splunkd pid=744] splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_under_systemd [process-runner
├─ 1224 /opt/splunk/bin/splunkd instrument-resource-usage -p 8089 --with-kvstore
├─ 1315 /opt/splunk/bin/python3.7 -O /opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/root.py --proxied=127.0.0.1,8065,8000
└─23617 /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_secure_gateway/bin/ssg_enable_modular_input.py

----------------------------------------------------------

contents of Splunkd.service:

#This unit file replaces the traditional start-up script for systemd
#configurations, and is used when enabling boot-start for Splunk on
#systemd-based Linux distributions.

[Unit]
Description=Systemd service file for Splunk, generated by 'splunk enable boot-start'
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
Restart=always
ExecStart=/opt/splunk/bin/splunk _internal_launch_under_systemd
KillMode=mixed
KillSignal=SIGINT
TimeoutStopSec=360
LimitNOFILE=65536
LimitNPROC=51200
LimitRTPRIO=99
SuccessExitStatus=51 52
RestartPreventExitStatus=51
RestartForceExitStatus=52
User=splunk
Group=splunk
Delegate=true
CPUShares=1024
MemoryLimit=8201170944
PermissionsStartOnly=true
ExecStartPost=-/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/cpu/system.slice/%n"
ExecStartPost=-/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/memory/system.slice/%n"

[Install]
WantedBy=multi-user.target

Re: Why is Splunk is restarting itself?

PickleRick — Mon, 06 Mar 2023 20:59:32 GMT

There are no miracles. If the splunk daemon process is starting "by itself", something must be causing it. Either you have some other form of service management software (monit?) or - much more probable - you have misconfigured your systemd and have multiple units defined for splunk. You can try to find them with

find /etc/systemd -iname \*splunk\*

and/or

find /etc/systemd | xargs grep splunk

Re: Why is Splunk is restarting itself?

ashraf_sj — Tue, 03 Oct 2023 15:58:35 GMT

Any updates on this, have the same situation

Re: Why is Splunk is restarting itself?

jason0 — Tue, 24 Feb 2026 16:56:31 GMT

Hello,

It's been a few years and I finally had time to figure this out. There was a SolarWinds agent that was periodically restarting splunk. I have no idea why, as we had stopped using SolarWinds a while ago. Uninstalling swiagent fixed it.

To find out what was restarting splunk I ran the following commands (centos 7):

PID=$(pgrep -n splunkd); echo $PID; systemctl status $PID --no-pager
PID=$(pgrep -n splunkd); cat /proc/$PID/cgroup

the output told me splunk was being started by SolarWinds.

Some more information:

PID=$(pgrep -n splunkd)
ps -o pid,ppid,cmd -p "$PID"
ps -fp "$(ps -o ppid= -p "$PID")"

This will show the parent relationship to the splunk process.

ChatGPT was helpful with the more obscure way to find what was starting splunk.

Re: Why is Splunk is restarting itself?

PickleRick — Tue, 24 Feb 2026 20:35:38 GMT

Ha! I knew there was something fishy going on. Splunk shouldn't just restart out of the blue.

Good you found it!