If it is RHEL 7.x + I would recommend using systemd to manage the splunk service: $SPLUNK_HOME/bin/splunk enable boot-start -systemd-managed 1 -user <username> -group <groupname> https://docs.splunk.com/Documentation/Splunk/8.1.0/Admin/RunSplunkassystemdservice This will create a file in /etc/systemd/system/Splunkd.service and I replace the contents of that with: [Unit]
After=network.target
[Service]
Type=simple
Restart=always
ExecStart= /opt/splunk/bin/splunk _internal_launch_under_systemd
ExecStop = /opt/splunk/bin/splunk stop
ExecReload = /opt/splunk/bin/splunk restart
LimitCORE = 0
LimitFSIZE = infinity
LimitDATA = infinity
LimitNPROC = 20480
LimitNOFILE = 65536
TimeoutSec = 300
SuccessExitStatus=51 52
RestartPreventExitStatus=51
RestartForceExitStatus=52
Delegate=true
MemoryLimit=100G
CPUShares=1024
PermissionsStartOnly=true
ExecStartPost=/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/cpu/system.slice/%n"
ExecStartPost=/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/memory/system.slice/%n"
[Install]
WantedBy=multi-user.target This file will handle required ulimit settings and deal with permissions per Splunk's env specs. Also note that I am running Splunk as the linux user splunk which you can change should you want to. If you choose to do this make sure you refresh systemd daemon: systemctl daemon-reload I prefer running Splunk using systemd instead of the legacy init.d as there is more functionality and I would assume it is going to be more supported moving forward.
... View more
Sometimes there's invisible whitespace in the numbers:
eval Avg=replace(Avg," ","")
Had this same issue - numbers appear as text so don't calculate or plot, especially when extracted with REX at search.
... View more