topic Re: Is there a systemd unit file for Splunk? in Splunk Search

Is there a systemd unit file for Splunk?

Yorokobi — Mon, 24 Sep 2012 20:00:50 GMT

systemd replaces SysV init scripts and some Linux distributions are migrating to or currently support systemd (such as Arch Linux). The traditional start-up script /etc/init.d/splunk (or /etc/rc.d/splunk) does not work in a pure systemd configuration, a unit file is required.

This thread is getting outdated. See the official documentation and additional information below:
- https://docs.splunk.com/Documentation/Splunk/7.2.6/Admin/RunSplunkassystemdservice
- https://docs.splunk.com/Documentation/Splunk/7.2.6/Workloads/Configuresystemd
- https://answers.splunk.com/answers/738877/splunk-systemd-unit-file-in-versions-722-and-newer.html
- https://www.duanewaddle.com/splunk-7-2-2-and-systemd/

Re: Is there a systemd unit file for Splunk?

Yorokobi — Mon, 24 Sep 2012 20:05:43 GMT

EDIT: I have tested this on Arch Linux and CentOS 7. I added the LimitNOFILE and PIDFile entries based on feedback from this Answers thread and conversations in both Splunk IRC and User Group Slack.

This unit file works for both Enterprise Splunk and the Universal Forwarder, simply change the description, paths, and unit file name as appropriate.

/etc/systemd/system/splunkd.service

[Unit]
Description=Splunk Enterprise 6.5.0
After=network.target
Wants=network.target

[Service]
Type=forking
RemainAfterExit=False
User=splunk
Group=splunk
LimitNOFILE=65536
ExecStart=/opt/splunk/bin/splunk start --accept-license --answer-yes --no-prompt
ExecStop=/opt/splunk/bin/splunk stop
PIDFile=/opt/splunk/var/run/splunk/splunkd.pid

[Install]
WantedBy=multi-user.target
# If you want to use $(systemctl [start|stop|restart] splunk) instead of splunkd ...
Alias=splunk.service

Re: Is there a systemd unit file for Splunk?

Yorokobi — Mon, 24 Sep 2012 20:56:47 GMT

To enable the unit file:
sudo systemctl enable splunkd

Re: Is there a systemd unit file for Splunk?

sspencer_splunk — Mon, 17 Dec 2012 00:53:46 GMT

According to systemd docs, variables cannot be used as the first argument of the ExecStart, ExecStop, and/or ExecReload options. Systemd will not expand those variables. My testing agrees with the documentation. I had to restore the absolute path as shown in the top entry of this answer.

Re: Is there a systemd unit file for Splunk?

sspencer_splunk — Mon, 17 Dec 2012 01:18:06 GMT

Here's something I threw together for a Universal Forwarder that's installed on a Fedora 17 endpoint. It should work on most *nix distros. There are a few assumptions I've made:

You have NOT previously enabled Splunk to start at boot by running the command "splunk enable boot-start" or performed some similar action to start Splunk automatically.
You are going to run the forwarder with root privileges. (Look up systemd.exec(5) man page for details on running the forwarder as a less-privileged user.)
Your forwarder is installed in /opt/splunkforwarder.
You will perform these commands below as root or with equivalent sudo privileges.

I started with Yorokobi's template above. Copy this text into a file named "/usr/lib/systemd/system/splunkforwarder.service".

[Unit]
Description=Splunk Universal Forwarder
Wants=network.target
After=network.target

[Service]
Type=forking
RemainAfterExit=yes
ExecStart=/opt/splunkforwarder/bin/splunk start
ExecStop=/opt/splunkforwarder/bin/splunk stop
ExecReload=/opt/splunkforwarder/bin/splunk restart
StandardOutput=syslog

[Install]
WantedBy=multi-user.target

Next, create a symlink that systemd will use to fetch this control file.

cd /etc/systemd/system/multi-user.target.wants
ln -sf /usr/lib/systemd/system/splunkforwarder.service splunkforwarder.service

That's it.

Next time you reboot your endpoint, the Splunk Forwarder should start automatically. You can now use systemctl to start and stop the Splunk forwarder at your leisure.

Re: Is there a systemd unit file for Splunk?

Yorokobi — Fri, 21 Dec 2012 21:46:54 GMT

Good catch, sspencer. I expect these answers will change over time as systemd evolves.

Re: Is there a systemd unit file for Splunk?

ephemeric — Wed, 06 May 2015 13:17:14 GMT

Don't need syslog as Splunk logs to splunkd.log and has user and group assigned.

One doesn't need the symlink as above, simply do:

systemctl enable splunkindexer.service

[Unit]
Description=Splunk indexer service
Wants=network.target
After=network.target

[Service]
Type=forking
RemainAfterExit=yes
Restart=on-failure
RestartSec=30s
User=splunk
Group=splunk
ExecStart=/opt/splunk/bin/splunk start
ExecStop=/opt/splunk/bin/splunk stop
ExecReload=/opt/splunk/bin/splunk restart
StandardOutput=null

[Install]
WantedBy=multi-user.target

Re: Is there a systemd unit file for Splunk?

mcs24 — Fri, 15 May 2015 09:54:52 GMT

ephemeric, you nailed the config. My two additions:

1. Use /usr/local for custom configs
Good sysadmin practice would be to put the config in /usr/local/lib rather than /usr/lib, since it isn't managed by the package--systemd supports this for that reason.

2. Daemon reload is required to read the new config before you can run a systemctl enable

My steps (as root) were the following:
(note that I am using the name "splunk.service" for service and file)

mkdir -p /usr/local/lib/systemd/system/
cp ~/MYCUSTOM.systemd.splunk.service /usr/local/lib/systemd/system/splunk.service
systemctl daemon-reload
systemctl enable splunk.service

Re: Is there a systemd unit file for Splunk?

FloydATC — Wed, 20 Jul 2016 12:09:29 GMT

From what I understand, custom made service files should ideally be placed in "/etc/systemd/system/" and not "/usr/lib/systemd/system/" because the latter should be touched only by package managers. Either place should work just fine in most cases but could cause issues if Splunk later on decides to ship their own service file (as I expect they will).

Re: Is there a systemd unit file for Splunk?

ephemeric — Thu, 11 Aug 2016 15:36:04 GMT

Thank you for the additions.

I see that RTFM says /etc/systemd/system/ but I suppose much of muchness.

Re: Is there a systemd unit file for Splunk?

ephemeric — Thu, 11 Aug 2016 16:20:08 GMT

Updated:

[Unit]
Description=Splunk service
Documentation=
After=network.target
Wants=network.target

[Service]
Type=forking
RemainAfterExit=yes
Restart=on-failure
RestartSec=30s
User=splunk
Group=splunk
LimitNOFILE=64000
ExecStart=/opt/splunk/bin/splunk start --accept-license --answer-yes --no-prompt
ExecStop=/opt/splunk/bin/splunk stop
ExecReload=/opt/splunk/bin/splunk restart

[Install]
WantedBy=multi-user.target

Re: Is there a systemd unit file for Splunk?

ephemeric — Thu, 11 Aug 2016 16:21:59 GMT

However the above does not restart if you kill a process.

Any ideas, anyone?

Thank you.

PS Why does Splunk STILL not ship with a service unit file?

Re: Is there a systemd unit file for Splunk?

joebisesi — Tue, 11 Oct 2016 20:09:15 GMT

Here is what I have configured that addresses restarting the process if the process is killed by anything except "systemctl stop splunk.service", this includes ./splunk stop. If you want to stop the splunk services and keep them from starting back up in 30s, you must issue this command: systemctl stop splunk.service

[Unit]
Description=Splunkd service
After=network.target
Wants=network.target

[Service]
Type=forking
RemainAfterExit=False
Restart=always
Restart=30s
User=splunk
Group=splunk
ExecStart=/opt/splunk/bin/splunk start
ExecStop=/opt/splunk/bin/splunk stop
Restart=/opt/splunk/bin/splunk restart

[Install]
WantedBy=multi-user.target

This is saved as /etc/systemd/system/splunk.service.
This was tested on CentOS 7
Hope this helps someone....

Re: Is there a systemd unit file for Splunk?

ephemeric — Wed, 12 Oct 2016 07:50:45 GMT

Hi,

Latest one I'm using, no need for RemainAfterexit:

"RemainAfterExit=
Takes a boolean value that specifies whether the service shall be considered active even when all its processes exited. Defaults to no."

[Unit]
Description=Splunk server service
Documentation=
After=network.target
Wants=network.target
[Service]
Type=forking
Restart=always
RestartSec=30s
User=splunk
Group=splunk
LimitNOFILE=64000
ExecStart=/opt/splunk/bin/splunk start --accept-license --answer-yes --no-prompt
ExecStop=/opt/splunk/bin/splunk stop
ExecReload=/opt/splunk/bin/splunk restart
[Install]
WantedBy=multi-user.target

Re: Is there a systemd unit file for Splunk?

joebisesi — Wed, 12 Oct 2016 10:54:45 GMT

Agreed on the "RemainAfterExit" defaulting to false, so there is not a need for it.
A couple of questions for you:

What is the "LimitNOFILE=64000" doing?
Is the Splunk server service restarting for you with the "ExecReload=/opt/../splunk restart"?

Re: Is there a systemd unit file for Splunk?

ephemeric — Wed, 12 Oct 2016 13:47:31 GMT

https://docs.splunk.com/Documentation/Splunk/6.5.0/Troubleshooting/ulimitErrors (Google "linux max file descriptors").
Yes, it does restart, you could also change to "ExecReload=/opt/splunk/bin/splunk reload [object]". Maybe somebody knows how to reload everything?

[root@template ~]# cat /etc/systemd/system/splunk.service
[Unit]
Description=Splunk service
Documentation=
After=network.target
Wants=network.target

[Service]
Type=forking
Restart=always
RestartSec=10s
User=splunk
Group=splunk
LimitNOFILE=64000
ExecStart=/opt/splunk/bin/splunk start --accept-license --answer-yes --no-prompt
ExecStop=/opt/splunk/bin/splunk stop
ExecReload=/opt/splunk/bin/splunk restart

[Install]
WantedBy=multi-user.target

[root@template ~]# systemctl status splunk.service
● splunk.service - Splunk service
Loaded: loaded (/etc/systemd/system/splunk.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2016-10-12 15:28:23 SAST; 8min ago
Process: 25379 ExecStop=/opt/splunk/bin/splunk stop (code=exited, status=0/SUCCESS)
Process: 25181 ExecReload=/opt/splunk/bin/splunk restart (code=exited, status=0/SUCCESS)
Process: 25404 ExecStart=/opt/splunk/bin/splunk start --accept-license --answer-yes --no-prompt (code=exited, status=0/SUCCESS)
Main PID: 25450 (splunkd)
CGroup: /system.slice/splunk.service
├─25450 splunkd -h 0.0.0.0 -p 8089 start
├─25455 [splunkd pid=25450] splunkd -h 0.0.0.0 -p 8089 start [process-runner]
├─25472 mongod --dbpath=/opt/splunk/var/lib/splunk/kvstore/mongo --port=8191 --timeStampFormat=iso8601-utc --smallfiles --oplogSize=200 --keyFile=/opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key --setParameter=enableLocalh...
├─25533 /opt/splunk/bin/python -O /opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/root.py --proxied=127.0.0.1,8065,8000
└─25574 /opt/splunk/bin/splunkd instrument-resource-usage -p 8089 --with-kvstore

Oct 12 15:28:21 template.ephemeric.local splunk[25404]: Done
Oct 12 15:28:21 template.ephemeric.local splunk[25404]: Checking default conf files for edits...
Oct 12 15:28:21 template.ephemeric.local splunk[25404]: Validating installed files against hashes from '/opt/splunk/splunk-6.5.0-59c8927def0f-linux-2.6-x86_64-manifest'
Oct 12 15:28:21 template.ephemeric.local splunk[25404]: [ OK ]
Oct 12 15:28:22 template.ephemeric.local splunk[25404]: All installed files intact.
Oct 12 15:28:22 template.ephemeric.local splunk[25404]: Done
Oct 12 15:28:22 template.ephemeric.local splunk[25404]: All preliminary checks passed.
Oct 12 15:28:22 template.ephemeric.local splunk[25404]: Starting splunk server daemon (splunkd)...
Oct 12 15:28:22 template.ephemeric.local splunk[25404]: Done
Oct 12 15:28:23 template.ephemeric.local systemd[1]: Started Splunk service.

[root@template ~]# systemctl reload splunk.service

Oct 12 15:37:02 template.ephemeric.local systemd[1]: Reloaded Splunk service.
[root@template ~]# systemctl status splunk.service
● splunk.service - Splunk service
Loaded: loaded (/etc/systemd/system/splunk.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2016-10-12 15:37:15 SAST; 14s ago
Process: 25855 ExecStop=/opt/splunk/bin/splunk stop (code=exited, status=0/SUCCESS)
Process: 25611 ExecReload=/opt/splunk/bin/splunk restart (code=exited, status=0/SUCCESS)
Process: 25881 ExecStart=/opt/splunk/bin/splunk start --accept-license --answer-yes --no-prompt (code=exited, status=0/SUCCESS)
Main PID: 25928 (splunkd)
CGroup: /system.slice/splunk.service
├─25928 splunkd -h 0.0.0.0 -p 8089 start
├─25932 [splunkd pid=25928] splunkd -h 0.0.0.0 -p 8089 start [process-runner]
├─25949 mongod --dbpath=/opt/splunk/var/lib/splunk/kvstore/mongo --port=8191 --timeStampFormat=iso8601-utc --smallfiles --oplogSize=200 --keyFile=/opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key --setParameter=enableLocalh...
├─25999 /opt/splunk/bin/python -O /opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/root.py --proxied=127.0.0.1,8065,8000
└─26049 /opt/splunk/bin/splunkd instrument-resource-usage -p 8089 --with-kvstore

Oct 12 15:37:14 template.ephemeric.local splunk[25881]: Done
Oct 12 15:37:14 template.ephemeric.local splunk[25881]: Checking default conf files for edits...
Oct 12 15:37:14 template.ephemeric.local splunk[25881]: Validating installed files against hashes from '/opt/splunk/splunk-6.5.0-59c8927def0f-linux-2.6-x86_64-manifest'
Oct 12 15:37:14 template.ephemeric.local splunk[25881]: [ OK ]
Oct 12 15:37:14 template.ephemeric.local splunk[25881]: All installed files intact.
Oct 12 15:37:14 template.ephemeric.local splunk[25881]: Done
Oct 12 15:37:14 template.ephemeric.local splunk[25881]: All preliminary checks passed.
Oct 12 15:37:14 template.ephemeric.local splunk[25881]: Starting splunk server daemon (splunkd)...
Oct 12 15:37:14 template.ephemeric.local splunk[25881]: Done
Oct 12 15:37:15 template.ephemeric.local systemd[1]: Started Splunk service.

[root@template ~]# ss -tnl
State Recv-Q Send-Q Local Address:Port Peer Address:Port

LISTEN 0 128 :22 *:

LISTEN 0 128 :8089 *:

LISTEN 0 100 127.0.0.1:25 :

LISTEN 0 128 :8191 *:

LISTEN 0 128 :8000 *:

LISTEN 0 100 127.0.0.1:8065 :

LISTEN 0 100 ::1:25 :::*

Re: Is there a systemd unit file for Splunk?

andrewfoglesong — Wed, 12 Apr 2017 18:24:11 GMT

You're follow up answers regarding the "Restart" and "RestartSec" options worked for me. Key point from my testing running CentOS7 and Splunk 6.5.2, DO NOT USE “RemainAfterExit=yes”. Your Main PID will remain the PID of your initial process start. When a Deployment Server, Cluster Master, or local user restarts Splunk, systemd will not track the new PID as the main one. Also, if Splunk crashes or is stopped by the user, systemd will still report “active” because it’s holding on to the successful start of that initial PID.

Re: Is there a systemd unit file for Splunk?

fenrisdacat — Thu, 15 Jun 2017 15:53:22 GMT

FloydATC is correct. Below is the from "man systemd.unit"

Table 1.  Load path when running in system mode (--system).
       ┌────────────────────────┬─────────────────────────────┐
       │Path                    │ Description                 │
       ├────────────────────────┼─────────────────────────────┤
       │/etc/systemd/system     │ Local configuration         │
       ├────────────────────────┼─────────────────────────────┤
       │/run/systemd/system     │ Runtime units               │
       ├────────────────────────┼─────────────────────────────┤
       │/usr/lib/systemd/system │ Units of installed packages │
       └────────────────────────┴─────────────────────────────┘

Re: Is there a systemd unit file for Splunk?

andreworg — Tue, 01 Aug 2017 10:08:41 GMT

how come the "splunk enable boot-start" has not been updated yet?
like it or not, systemd has been standard in most distros for months now.

Re: Is there a systemd unit file for Splunk?

mattymo — Wed, 23 Aug 2017 04:15:39 GMT

+1, working nicely in my lab