Is there a systemd unit file for Splunk?

[Yorokobi]

systemd replaces SysV init scripts and some Linux distributions are migrating to or currently support systemd (such as Arch Linux). The traditional start-up script /etc/init.d/splunk (or /etc/rc.d/splunk) does not work in a pure systemd configuration, a unit file is required.

This thread is getting outdated. See the official documentation and additional information below:
- https://docs.splunk.com/Documentation/Splunk/7.2.6/Admin/RunSplunkassystemdservice
- https://docs.splunk.com/Documentation/Splunk/7.2.6/Workloads/Configuresystemd
- https://answers.splunk.com/answers/738877/splunk-systemd-unit-file-in-versions-722-and-newer.html
- https://www.duanewaddle.com/splunk-7-2-2-and-systemd/

[andreworg]

how come the "splunk enable boot-start" has not been updated yet?
like it or not, systemd has been standard in most distros for months now.

[ephemeric]

Hi,

Latest one I'm using, no need for RemainAfterexit:

"RemainAfterExit=
Takes a boolean value that specifies whether the service shall be considered active even when all its processes exited. Defaults to no."

[Unit]
Description=Splunk server service
Documentation=
After=network.target
Wants=network.target
[Service]
Type=forking
Restart=always
RestartSec=30s
User=splunk
Group=splunk
LimitNOFILE=64000
ExecStart=/opt/splunk/bin/splunk start --accept-license --answer-yes --no-prompt
ExecStop=/opt/splunk/bin/splunk stop
ExecReload=/opt/splunk/bin/splunk restart
[Install]
WantedBy=multi-user.target

[mattymo]

+1, working nicely in my lab

[ephemeric]

1. https://docs.splunk.com/Documentation/Splunk/6.5.0/Troubleshooting/ulimitErrors (Google "linux max file descriptors").

2. Yes, it does restart, you could also change to "ExecReload=/opt/splunk/bin/splunk reload [object]". Maybe somebody knows how to reload everything?

[root@template ~]# cat /etc/systemd/system/splunk.service
[Unit]
Description=Splunk service
Documentation=
After=network.target
Wants=network.target

[Service]
Type=forking
Restart=always
RestartSec=10s
User=splunk
Group=splunk
LimitNOFILE=64000
ExecStart=/opt/splunk/bin/splunk start --accept-license --answer-yes --no-prompt
ExecStop=/opt/splunk/bin/splunk stop
ExecReload=/opt/splunk/bin/splunk restart

[Install]
WantedBy=multi-user.target

[root@template ~]# systemctl status splunk.service
● splunk.service - Splunk service
Loaded: loaded (/etc/systemd/system/splunk.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2016-10-12 15:28:23 SAST; 8min ago
Process: 25379 ExecStop=/opt/splunk/bin/splunk stop (code=exited, status=0/SUCCESS)
Process: 25181 ExecReload=/opt/splunk/bin/splunk restart (code=exited, status=0/SUCCESS)
Process: 25404 ExecStart=/opt/splunk/bin/splunk start --accept-license --answer-yes --no-prompt (code=exited, status=0/SUCCESS)
Main PID: 25450 (splunkd)
CGroup: /system.slice/splunk.service
├─25450 splunkd -h 0.0.0.0 -p 8089 start
├─25455 [splunkd pid=25450] splunkd -h 0.0.0.0 -p 8089 start [process-runner]
├─25472 mongod --dbpath=/opt/splunk/var/lib/splunk/kvstore/mongo --port=8191 --timeStampFormat=iso8601-utc --smallfiles --oplogSize=200 --keyFile=/opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key --setParameter=enableLocalh...
├─25533 /opt/splunk/bin/python -O /opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/root.py --proxied=127.0.0.1,8065,8000
└─25574 /opt/splunk/bin/splunkd instrument-resource-usage -p 8089 --with-kvstore

Oct 12 15:28:21 template.ephemeric.local splunk[25404]: Done
Oct 12 15:28:21 template.ephemeric.local splunk[25404]: Checking default conf files for edits...
Oct 12 15:28:21 template.ephemeric.local splunk[25404]: Validating installed files against hashes from '/opt/splunk/splunk-6.5.0-59c8927def0f-linux-2.6-x86_64-manifest'
Oct 12 15:28:21 template.ephemeric.local splunk[25404]: [ OK ]
Oct 12 15:28:22 template.ephemeric.local splunk[25404]: All installed files intact.
Oct 12 15:28:22 template.ephemeric.local splunk[25404]: Done
Oct 12 15:28:22 template.ephemeric.local splunk[25404]: All preliminary checks passed.
Oct 12 15:28:22 template.ephemeric.local splunk[25404]: Starting splunk server daemon (splunkd)...
Oct 12 15:28:22 template.ephemeric.local splunk[25404]: Done
Oct 12 15:28:23 template.ephemeric.local systemd[1]: Started Splunk service.

[root@template ~]# systemctl reload splunk.service

[root@template ~]# systemctl status splunk.service
● splunk.service - Splunk service
Loaded: loaded (/etc/systemd/system/splunk.service; enabled; vendor preset: disabled)
Active: activating (auto-restart) since Wed 2016-10-12 15:37:02 SAST; 1s ago
Process: 25855 ExecStop=/opt/splunk/bin/splunk stop (code=exited, status=0/SUCCESS)
Process: 25611 ExecReload=/opt/splunk/bin/splunk restart (code=exited, status=0/SUCCESS)
Process: 25404 ExecStart=/opt/splunk/bin/splunk start --accept-license --answer-yes --no-prompt (code=exited, status=0/SUCCESS)
Main PID: 25450 (code=exited, status=0/SUCCESS)

Oct 12 15:37:02 template.ephemeric.local systemd[1]: Reloaded Splunk service.
[root@template ~]# systemctl status splunk.service
● splunk.service - Splunk service
Loaded: loaded (/etc/systemd/system/splunk.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2016-10-12 15:37:15 SAST; 14s ago
Process: 25855 ExecStop=/opt/splunk/bin/splunk stop (code=exited, status=0/SUCCESS)
Process: 25611 ExecReload=/opt/splunk/bin/splunk restart (code=exited, status=0/SUCCESS)
Process: 25881 ExecStart=/opt/splunk/bin/splunk start --accept-license --answer-yes --no-prompt (code=exited, status=0/SUCCESS)
Main PID: 25928 (splunkd)
CGroup: /system.slice/splunk.service
├─25928 splunkd -h 0.0.0.0 -p 8089 start
├─25932 [splunkd pid=25928] splunkd -h 0.0.0.0 -p 8089 start [process-runner]
├─25949 mongod --dbpath=/opt/splunk/var/lib/splunk/kvstore/mongo --port=8191 --timeStampFormat=iso8601-utc --smallfiles --oplogSize=200 --keyFile=/opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key --setParameter=enableLocalh...
├─25999 /opt/splunk/bin/python -O /opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/root.py --proxied=127.0.0.1,8065,8000
└─26049 /opt/splunk/bin/splunkd instrument-resource-usage -p 8089 --with-kvstore

Oct 12 15:37:14 template.ephemeric.local splunk[25881]: Done
Oct 12 15:37:14 template.ephemeric.local splunk[25881]: Checking default conf files for edits...
Oct 12 15:37:14 template.ephemeric.local splunk[25881]: Validating installed files against hashes from '/opt/splunk/splunk-6.5.0-59c8927def0f-linux-2.6-x86_64-manifest'
Oct 12 15:37:14 template.ephemeric.local splunk[25881]: [ OK ]
Oct 12 15:37:14 template.ephemeric.local splunk[25881]: All installed files intact.
Oct 12 15:37:14 template.ephemeric.local splunk[25881]: Done
Oct 12 15:37:14 template.ephemeric.local splunk[25881]: All preliminary checks passed.
Oct 12 15:37:14 template.ephemeric.local splunk[25881]: Starting splunk server daemon (splunkd)...
Oct 12 15:37:14 template.ephemeric.local splunk[25881]: Done
Oct 12 15:37:15 template.ephemeric.local systemd[1]: Started Splunk service.

[root@template ~]# ss -tnl
State Recv-Q Send-Q Local Address:Port Peer Address:Port

LISTEN 0 128 :22 *:

LISTEN 0 128 :8089 *:

LISTEN 0 100 127.0.0.1:25 :

LISTEN 0 128 :8191 *:

LISTEN 0 128 :8000 *:

LISTEN 0 100 127.0.0.1:8065 :

LISTEN 0 100 ::1:25 :::*

[joebisesi]

Agreed on the "RemainAfterExit" defaulting to false, so there is not a need for it.
A couple of questions for you:

1. What is the "LimitNOFILE=64000" doing?
2. Is the Splunk server service restarting for you with the "ExecReload=/opt/../splunk restart"?

[twinspop]

On the latest RHEL, when I run with RemainAfterExit set to false/no, internally generated restarts fail. Splukn just stops. Example, accessing the REST endpoint for system reload. Or if a deployment server delivers an app that is flagged for restart. I end up with no service running. If I set RemainAfterExit to true/yes, systemd completely loses track of splunk after any stop or internally generated restart. Neither situation is optimal.

[joebisesi]

Can you provide a copy of your splunk.service file? Are you running splunk as root or as a different user? Is this issue only happening with the two scenarios that you listed?

[twinspop]

Below is my template i was using. Since this post, I changed the Restart value to always. EDIT: Yes, normally the User value is set to root. (Working on correcting that.)

[Unit]
Description=Splunk Forwarder
After=network.target
Wants=network.target

[Service]
Type=forking
RemainAfterExit=False
User=??SPLUNK_USER??
LimitNOFILE=12000
ExecStart=??SPLUNK_INSTALL_DIR??/splunkforwarder/bin/splunk restart --accept-license --answer-yes --no-prompt
ExecStop=??SPLUNK_INSTALL_DIR??/splunkforwarder/bin/splunk stop
Restart=on-failure
RestartSec=20

[Install]
WantedBy=multi-user.target

[joebisesi]

I would try copying what I have entered in this post and see if it works. The differences would be the user, group and your splunk home directory.

[joebisesi]

Here is what I have configured that addresses restarting the process if the process is killed by anything except "systemctl stop splunk.service", this includes ./splunk stop. If you want to stop the splunk services and keep them from starting back up in 30s, you must issue this command: systemctl stop splunk.service

[Unit]
Description=Splunkd service
After=network.target
Wants=network.target

[Service]
Type=forking
RemainAfterExit=False
Restart=always
Restart=30s
User=splunk
Group=splunk
ExecStart=/opt/splunk/bin/splunk start
ExecStop=/opt/splunk/bin/splunk stop
Restart=/opt/splunk/bin/splunk restart

[Install]
WantedBy=multi-user.target

This is saved as /etc/systemd/system/splunk.service.
This was tested on CentOS 7
Hope this helps someone....

[FloydATC]

From what I understand, custom made service files should ideally be placed in "/etc/systemd/system/" and not "/usr/lib/systemd/system/" because the latter should be touched only by package managers. Either place should work just fine in most cases but could cause issues if Splunk later on decides to ship their own service file (as I expect they will).

[fenrisdacat]

FloydATC is correct. Below is the from "man systemd.unit"

Table 1.  Load path when running in system mode (--system).
       ┌────────────────────────┬─────────────────────────────┐
       │Path                    │ Description                 │
       ├────────────────────────┼─────────────────────────────┤
       │/etc/systemd/system     │ Local configuration         │
       ├────────────────────────┼─────────────────────────────┤
       │/run/systemd/system     │ Runtime units               │
       ├────────────────────────┼─────────────────────────────┤
       │/usr/lib/systemd/system │ Units of installed packages │
       └────────────────────────┴─────────────────────────────┘

[sspencer_splunk]

Here's something I threw together for a Universal Forwarder that's installed on a Fedora 17 endpoint. It should work on most *nix distros. There are a few assumptions I've made:

• You have NOT previously enabled Splunk to start at boot by running the command "splunk enable boot-start" or performed some similar action to start Splunk automatically.
• You are going to run the forwarder with root privileges. (Look up systemd.exec(5) man page for details on running the forwarder as a less-privileged user.)
• Your forwarder is installed in /opt/splunkforwarder.
• You will perform these commands below as root or with equivalent sudo privileges.

---

I started with Yorokobi's template above. Copy this text into a file named "/usr/lib/systemd/system/splunkforwarder.service".

[Unit]
Description=Splunk Universal Forwarder
Wants=network.target
After=network.target

[Service]
Type=forking
RemainAfterExit=yes
ExecStart=/opt/splunkforwarder/bin/splunk start
ExecStop=/opt/splunkforwarder/bin/splunk stop
ExecReload=/opt/splunkforwarder/bin/splunk restart
StandardOutput=syslog

[Install]
WantedBy=multi-user.target

Next, create a symlink that systemd will use to fetch this control file.

cd /etc/systemd/system/multi-user.target.wants
ln -sf /usr/lib/systemd/system/splunkforwarder.service splunkforwarder.service

That's it.

Next time you reboot your endpoint, the Splunk Forwarder should start automatically. You can now use systemctl to start and stop the Splunk forwarder at your leisure.

[jethompson_splu]

@sspencer_splunk -- Although you can place unit files in: /usr/lib/systemd/system --> This should not be manually done. This is the location that the unit file would be created via a Packaged install. This should also not be done unless provided by a Package Install due to the fact that this location can be "Moved, Modified, Removed, or Changed" by a Later OS Update and as such should not be used for this Purpose.

A manually created Services File for SystemD should be in: /etc/systemd/system/ and as such there is no reason for creating a Symlink for this process. This is also considered "Best Practice" in the Linux Community and is where I recommend placing any Manually Created Unit file.

[ephemeric]

Don't need syslog as Splunk logs to splunkd.log and has user and group assigned.

One doesn't need the symlink as above, simply do:

systemctl enable splunkindexer.service

[Unit]
Description=Splunk indexer service
Wants=network.target
After=network.target

[Service]
Type=forking
RemainAfterExit=yes
Restart=on-failure
RestartSec=30s
User=splunk
Group=splunk
ExecStart=/opt/splunk/bin/splunk start
ExecStop=/opt/splunk/bin/splunk stop
ExecReload=/opt/splunk/bin/splunk restart
StandardOutput=null

[Install]
WantedBy=multi-user.target

[ephemeric]

Updated:

[Unit]
Description=Splunk service
Documentation=
After=network.target
Wants=network.target

[Service]
Type=forking
RemainAfterExit=yes
Restart=on-failure
RestartSec=30s
User=splunk
Group=splunk
LimitNOFILE=64000
ExecStart=/opt/splunk/bin/splunk start --accept-license --answer-yes --no-prompt
ExecStop=/opt/splunk/bin/splunk stop
ExecReload=/opt/splunk/bin/splunk restart

[Install]
WantedBy=multi-user.target

[ephemeric]

However the above does not restart if you kill a process.

Any ideas, anyone?

Thank you.

PS Why does Splunk STILL not ship with a service unit file?

[andrewfoglesong]

You're follow up answers regarding the "Restart" and "RestartSec" options worked for me. Key point from my testing running CentOS7 and Splunk 6.5.2, DO NOT USE “RemainAfterExit=yes”. Your Main PID will remain the PID of your initial process start. When a Deployment Server, Cluster Master, or local user restarts Splunk, systemd will not track the new PID as the main one. Also, if Splunk crashes or is stopped by the user, systemd will still report “active” because it’s holding on to the successful start of that initial PID.

[mcs24]

ephemeric, you nailed the config. My two additions:

1. Use /usr/local for custom configs
Good sysadmin practice would be to put the config in /usr/local/lib rather than /usr/lib, since it isn't managed by the package--systemd supports this for that reason.

2. Daemon reload is required to read the new config before you can run a systemctl enable

My steps (as root) were the following:
(note that I am using the name "splunk.service" for service and file)

mkdir -p /usr/local/lib/systemd/system/
cp ~/MYCUSTOM.systemd.splunk.service /usr/local/lib/systemd/system/splunk.service
systemctl daemon-reload
systemctl enable splunk.service

[ephemeric]

Thank you for the additions.

I see that RTFM says /etc/systemd/system/ but I suppose much of muchness.

[Yorokobi]

EDIT: I have tested this on Arch Linux and CentOS 7. I added the LimitNOFILE and PIDFile entries based on feedback from this Answers thread and conversations in both Splunk IRC and User Group Slack.

This unit file works for both Enterprise Splunk and the Universal Forwarder, simply change the description, paths, and unit file name as appropriate.

/etc/systemd/system/splunkd.service

[Unit]
Description=Splunk Enterprise 6.5.0
After=network.target
Wants=network.target

[Service]
Type=forking
RemainAfterExit=False
User=splunk
Group=splunk
LimitNOFILE=65536
ExecStart=/opt/splunk/bin/splunk start --accept-license --answer-yes --no-prompt
ExecStop=/opt/splunk/bin/splunk stop
PIDFile=/opt/splunk/var/run/splunk/splunkd.pid

[Install]
WantedBy=multi-user.target
# If you want to use $(systemctl [start|stop|restart] splunk) instead of splunkd ...
Alias=splunk.service