Splunk Enterprise

Why is Splunk is restarting itself?

jason0
Path Finder

Hello, 

I have a splunk heavy forwarder (splunk 9.0.0.1, centos 7) configured as a heavy forwarder.  When I issue "splunk stop" as either the splunk user or user root, it will restart within a minute.

This system has been configured to start as a systemd service and user splunk is allowed to issue systemctl commands to stop and restart.

just now, I completely disabled the systemd service and then rebooted.  After about a minute of being fully booted, splunk automatically started up.  Its as if there's a parasitic cron or anacron job that starts it if its not running.

What could be causing this?

I would really rather splunk stayed down especially since I am usually altering the configuration files at the time and it starts up before I can complete the task.

--jason

Labels (2)

ashraf_sj
Explorer

Any updates on this, have the same situation

0 Karma

PickleRick
SplunkTrust
SplunkTrust

There are no miracles. If the splunk daemon process is starting "by itself", something must be causing it. Either you have some other form of service management software (monit?) or - much more probable - you have misconfigured your systemd and have multiple units defined for splunk. You can try to find them with

find /etc/systemd -iname \*splunk\*

and/or

find /etc/systemd | xargs grep splunk
0 Karma

jason0
Path Finder

First, I can indeed confirm I entered "systemctl disable Splunkd"

Here is the rest:

--------------------------------

systemctl status Splunkd


● Splunkd.service - Systemd service file for Splunk, generated by 'splunk enable boot-start'
Loaded: loaded (/etc/systemd/system/Splunkd.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2023-03-03 13:41:35 PST; 2 days ago
Process: 748 ExecStartPost=/bin/bash -c chown -R splunk:splunk /sys/fs/cgroup/memory/system.slice/%n (code=exited, status=0/SUCCESS)
Process: 745 ExecStartPost=/bin/bash -c chown -R splunk:splunk /sys/fs/cgroup/cpu/system.slice/%n (code=exited, status=0/SUCCESS)
Main PID: 744 (splunkd)
Memory: 1.0G (limit: 7.6G)
CGroup: /system.slice/Splunkd.service
├─ 744 splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_under_systemd
├─ 889 [splunkd pid=744] splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_und... ├─ 1224 /opt/splunk/bin/splunkd instrument-resource-usage -p 8089 --with-kvstore
├─ 1315 /opt/splunk/bin/python3.7 -O /opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrspark... └─23617 /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_secure_gateway/bin/ssg_enable_modular_i...splunk@slsplunksyslog003p:~$ systemctl status Splunkd -l
● Splunkd.service - Systemd service file for Splunk, generated by 'splunk enable boot-start'
Loaded: loaded (/etc/systemd/system/Splunkd.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2023-03-03 13:41:35 PST; 2 days ago
Process: 748 ExecStartPost=/bin/bash -c chown -R splunk:splunk /sys/fs/cgroup/memory/system.slice/%n (code=exited, status=0/SUCCESS)
Process: 745 ExecStartPost=/bin/bash -c chown -R splunk:splunk /sys/fs/cgroup/cpu/system.slice/%n (code=exited, status=0/SUCCESS)
Main PID: 744 (splunkd)
Memory: 1.0G (limit: 7.6G)
CGroup: /system.slice/Splunkd.service
├─ 744 splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_under_systemd
├─ 889 [splunkd pid=744] splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_under_systemd [process-runner
├─ 1224 /opt/splunk/bin/splunkd instrument-resource-usage -p 8089 --with-kvstore
├─ 1315 /opt/splunk/bin/python3.7 -O /opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/root.py --proxied=127.0.0.1,8065,8000
└─23617 /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_secure_gateway/bin/ssg_enable_modular_input.py

 

----------------------------------------------------------

contents of Splunkd.service:

#This unit file replaces the traditional start-up script for systemd
#configurations, and is used when enabling boot-start for Splunk on
#systemd-based Linux distributions.

[Unit]
Description=Systemd service file for Splunk, generated by 'splunk enable boot-start'
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
Restart=always
ExecStart=/opt/splunk/bin/splunk _internal_launch_under_systemd
KillMode=mixed
KillSignal=SIGINT
TimeoutStopSec=360
LimitNOFILE=65536
LimitNPROC=51200
LimitRTPRIO=99
SuccessExitStatus=51 52
RestartPreventExitStatus=51
RestartForceExitStatus=52
User=splunk
Group=splunk
Delegate=true
CPUShares=1024
MemoryLimit=8201170944
PermissionsStartOnly=true
ExecStartPost=-/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/cpu/system.slice/%n"
ExecStartPost=-/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/memory/system.slice/%n"

[Install]
WantedBy=multi-user.target

 

0 Karma

Tom_Lundie
Contributor

Could you have an initd script starting Splunk?

ls /etc/init.d/

 

0 Karma

jason0
Path Finder

No.  the only files are:

  • /etc/systemd/system/splunkd.service
  • /etc/polkit-1/rules.d/10-Splunkd.rules

and there are no SYSV init files that start splunk.

0 Karma

Tom_Lundie
Contributor

Interesting, can you please share the output of:

systemctl status splunkd

 and

cat /etc/systemd/system/splunkd.service

Also just to confirm, to disable splunk you ran:

systemctl disable splunkd
0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

Industry Solutions for Supply Chain and OT, Amazon Use Cases, Plus More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Enterprise Security Content Update (ESCU) | New Releases

In November, the Splunk Threat Research Team had one release of new security content via the Enterprise ...