Is there a systemd unit file for Splunk?

[Yorokobi]

systemd replaces SysV init scripts and some Linux distributions are migrating to or currently support systemd (such as Arch Linux). The traditional start-up script /etc/init.d/splunk (or /etc/rc.d/splunk) does not work in a pure systemd configuration, a unit file is required.

This thread is getting outdated. See the official documentation and additional information below:
- https://docs.splunk.com/Documentation/Splunk/7.2.6/Admin/RunSplunkassystemdservice
- https://docs.splunk.com/Documentation/Splunk/7.2.6/Workloads/Configuresystemd
- https://answers.splunk.com/answers/738877/splunk-systemd-unit-file-in-versions-722-and-newer.html
- https://www.duanewaddle.com/splunk-7-2-2-and-systemd/

[twinspop]

Some systemd installs will use init style start-up scripts. Some don't. Splunk does not officially support "pure" systemd startup.

[jethompson_splu]

@compyy23 --> Yes, Splunk will still function properly with the "Old" Init.d Script that is provided from the "./splunk enable boot-start -user splunk" Command set. You can generate your own SystemD unit file (I have personally be using the one I posted for almost 2 years without any issues with Splunk 6.5, 6.6 & 7.0).

You will want to make sure that you issue the following to ensure Splunk is set for Boot-Time Start:

systemctl enable splunk.service --> this is similar to the Old "chkconfig"

@twinspop -- Although Splunk 7.1 and prior do not provide a SystemD Unit file for use, SystemD allows for the "Old" SysV init.d Script functionality and as such this is a Viable "Workaround".

As stated by @gjanders as of Splunk 7.2 a SystemD unit file will be created for a Linux Instance that is actually using SystemD.

Now as for a SystemD Unit file there were MANY Applications (Apache, Nginx, Clam-AV, and Many Legacy Systems still have not been updated) that have only recently updated their Software to be "SystemD Compliant" with a unit file for its Startup Process. If you create a SystemD unit file for your Splunk installation (Splunk 7.1 and Prior) then you will have a "Pure SystemD" installation of Splunk as you have stated.

[twinspop]

@jethompson - you say that like it's a universal truth. The company I work for provides a "secure build" of SLES linux. Init.d does not function. Getting systemd was required. And running it as non-root user required the solution I posted below.

[gjanders]

@twinspop, Splunk 7.2.0 now supports systemd officially refer to Configure systemd distributions

[twinspop]

Hmm. Interesting that it's buried in WLM config docs. One issue I haven't yet found a solution to: Splunk recognizes that it's configured with systemd. All stop|start|restart calls are now redirected to sysctl calls. But those calls require root. So if you're running as non-priv user, as you should, a restart command requires root password. Even if I allow sudo use for splunk commands, a root password prompt still fires.

[jethompson_splu]

@twinspop -- I wanted to make sure that there is not a Typo in your post as you state using the command: sysctl -> this is a ROOT only Command set as you are modifying Kernel Settings and Behavior with this Command Set.

You should be using: systemctl

You can provide a User with only Permissions to issue the SystemCTL Commands by using the sudoers file. You do not have to provide Full Sudo Permissions to a user. You can lock this down to a Single Command like: LS or you can have Multiple Commands like: sar, ls, ps, or anything really.

You can also setup this User Account to not be required to issue a Password when passing the command/command set that you configure inside of the Sudoers File. The following will provide some insights on how you can complete this type of "Restricted" Command usage for the Sudoers File:

https://www.cyberciti.biz/tips/allow-a-normal-user-to-run-commands-as-root.html
OR
https://www.atrixnet.com/allow-an-unprivileged-user-to-run-a-certain-command-with-sudo/

I personally use this method for my Splunk User account to only have the needed permissions in my CentOS 7.x Environment to issue:

sudo systemctl start splunk.service
sudo systemctl stop splunk.service
sudo systemctl restart splunk.service
sudo systemctl status splunk.service

[twinspop]

Step 2 on the linked docs: The name of the unit (.service) file should match exactly the name of the SPLUNK_SERVER_NAME value in splunk-launch.conf. So unless that's been changed from 'Splunkd',jethompson's commands above reference a config out of spec. Perhaps that's the trigger that makes Spunk intercept calls and redirect them to systemctl. Perhaps it works without it and no harm will be done.

[jethompson_splu]

@twinspop -- So to address your "out-of-spec" response. First off a Linux SystemD Unit file can literally be named anything, so if you wanted to name the Splunk Unit file: logranger.service You can do so. This would just mean that you need to call the service using: systemctl start/stop/restart logranger.service.

Yes the Splunk Configuration File: splunk-launch.conf uses splunkd (Or Splunkd) as the Splunk Service Name when it is running and this is just a "Best Practice" in Linux when the "Service" that is running is running as a Daemon at the System Level and as Splunk in this Case is considered a Daemon you would add the d to the End of the Process Name to identify that.

Now my posts with Commands for testing are example and may need to be updated based on your environment and situation. This will be the case almost 99% of the time as there is no way for anyone other than you to know how your Environment is configured and as such any Command Set provided is provided as an Example and again may need to be modified per situation.

Now with that being said, have you requested the Docs Team update the Document in question? From my understanding the Doc provided is providing a Sample Unit File and may need to be adjusted accordingly for your Environment.

I would also like to know if you are using cgroups inside of your Environment as that is not something that is set by default in some Environments and could be the direct reason that you experienced the issue that you have.

As I advised I have been using the Unit File that I provided in a previous post for a year without any issues and my Splunk user can Start/Stop/Restart Splunk using the following (this is in 1 of my testing environments and NOT using Linux Best Practices due to the fact that its a Test Environment):

sudo systemctl start/stop/restart splunk.service

Again Yes, I have not followed Linux "Best Practices" when I named that Specific Splunk Until File, now this is also on a System that is running 3 different Splunk Processes (SearchHead, Forwarder, Deployment Server -> All running on Same Host with Different Splunk Installations) . The Command set that I provided above is for the SearchHead Process that I have running on that Host.

I would also like to know if you are using Splunk 7.2 or if you are trying to use that Unit File for a different version of Splunk. If you are this could lead to potential issues as Splunk 7.1 and Prior was not directly coded for SystemD and as such there are potential issues that could arise by trying to use a "Pure" systemD Unit file for those Prior versions of Splunk.

I have tested the Unit file that I am using with the following versions of Splunk in my CentOS 7.4/7.5 Environments: 6.4.x, 6.5.x,. 6.6.x, 7.1.x

[twinspop]

My original comment was talking directly to the provided link showing Splunk's first recommended unit file. Following those directions will substitute systemctl commands for splunk commands that splunk admins have relied on since inception (2007 for me). And it breaks things in possibly unexpected ways. That was my only point. I think it's still valid.

[gjanders]

Please request the docs team to update the page in question once the case is resolved, thanks!
That way others including myself can avoid the issue 🙂

[joebisesi]

I agree with 'jethompson'. As you can see from my earlier post, I have been using this for two years now, and have had zero issues. The only thing I can add, is if splunk doesn't start using systemctl, you need to start it from the bin folder. Most likely there is a config file that is not configured correctly. The only way, that I have found, to discover the misconfiguration is to start splunk from the bin folder.

[gjanders]

@joebisesi define "works", have you tried restarting Splunk via the bin/splunk restart?
And if so does systemctl know that Splunk is running?

And when you shutdown does Splunk stop with the OS or does it get terminated?
I did extensive testing and logged a support case with Splunk about RHEL 7.x (I think it was 7.3) which is systemd based, while the Splunk process is stopped correctly when started with either systemd or init.d at boot time (where init.d is just a systemd service in this version), stopping only works if you have not restarted Splunk from the command line.

Once you restart Splunk from the command line systemd assumes Splunk is "dead" and does not stop it on shutdown, it terminates it via a kill signal.

7.2 is hopefully the fix for systemd based Linux systems pending the updates from twinspop/support.

[twinspop]

The sample unit file they posted under the WLM docs was only posted with 7.2.0. If you follow it exactly, it doesn't work with non-priveleged users. I've opened a case with support and they confirmed as much.

EDIT - it doesn't work meaning star|stop|restart controls require root level access

[twinspop]

Yes, i meant systemctl.

# sudo splunk set-up
splunk ALL=(ALL) NOPASSWD: /app/splunk/bin/splunk restart, /app/splunk/bin/splunk stop, /app/splunk/bin/splunk start

splunk@pocsh:~> sudo /app/splunk/bin/splunk restart
Stopping splunkd...
Shutting down.  Please wait, as this may take a few minutes.
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to stop 'Splunkd.service'.
Authenticating as: root
Password:

Using your method of instead allowing the raw systemctl command appears to work, tho.

[jethompson_splu]

@twinspop -- you are actually starting Splunk as Root when issuing the Command that you are issuing.

So what is most likely happening in your Environment is the following:

1) Splunk is actually being started as Root with the Command set you have issued.  The Sudo Command is not the SystemCTL Command. 
2) As Splunk was started as Root there is a "Permissions" Issue with your Splunk installation when you attempt to start Splunk as ANY user other than ROOT

To correct this issue you will need to complete the following:

1) DO NOT ISSUE the Command: sudo /app/splunk/bin/splunk start 
2) Change Ownership of ALL Splunk Files back to the Splunk User using:
chown -R splunk. /app/splunk
3) To start Splunk as the Splunk User issue the following:
/app/splunk/bin/splunk start

If there are Any Failures during the Startup Process for Splunk you will see this in the printout that is provided with Splunk starting. If you do run into an issue you may need to double check your SELinux Settings and make sure that SELinux has been disabled or you have the needed SELinux Context Rules in place for Splunk to Function properly.

Its recommended to Disable SELinux when possible.

[twinspop]

If (IF) you follow the page directions linked above (7.2.0 WLM featureset), you will end up where, as the non-priveleged splunk user, you can no longer run the splunk start|stop|restart commands. They are intercepted, running as systemctl commands, and run as root. But the systemctl subsystem asks for a password, whether or not you run it with sudo. As long as you have splunk-launch.conf set properly, you can start splunk as root. It will switch.

If you're not running the aforementioned sample systemd unit file, ¯\(ツ)/¯ your results will be different.

[jethompson_splu]

Also with SystemD you can actually get further information if there is a Service Failure by reviewing the JournalCTL of the Service. This if I recall correctly is seen using the following command set:

journalctl -xe

It's been a little bit since I have had to review a SystemD Service Failure so I might be a little wrong with the command set provided and as such you may need to review the Man Pages for journalctl to validate reviewing a "Service Failure Journal Log Entry".

[gjanders]

Also refer to the other systemd question/answer https://answers.splunk.com/answers/590209/rhelcentos-7-systemd-not-honoring-ulimits.html

[mandrewjeski]

line #1 is a path and will cause an error (RHEL7). you'll be able to start a forwarder but not enable it. you'll have other niggly little messages from systemd as well.

just remove it

[jethompson_splu]

So a while back I had worked with a customer that was wanting to make sure that SystemCTL and SystemD were able to Track the Splunk PID after a Restart issued from a Deploy/Deployment Server, as this would cause SystemCTL to "Lose" the Splunk PID and "freak out".

The following is the Unit File that I created for my Splunk Repro Environment in an attempt to assist the customer I had worked with:

 [Unit]
 Description=Splunk indexer service
 Wants=network.target
 After=network.target
 Requires=thp-disable.service

 [Service]
 Type=forking
 Restart=on-failure
 User=splunk
 Group=splunk
 ExecStart=/opt/splunk/bin/splunk start
 ExecStop=/opt/splunk/bin/splunk stop
 ExecReload=/opt/splunk/bin/splunk restart
 StandardOutput=syslog
 LimitNOFILE=65535
 LimitNPROC=16384
 LimitDATA=85256585
 LimitFSIZE=632524157485
 TimeoutSec=300
 PIDFile=/opt/splunk/var/run/splunk/splunkd.pid

 [Install]
 WantedBy=multi-user.target

The reason for the 300 Second Timeout is that is the "default" timeout for the Splunk WebUI to be made available for Splunk. This does not mean that Splunk has failed to start as there are times that the WebUI takes longer to start then expected. I also use a "Service Call" to disable THP for the Splunk Service as I have a couple of processes that can use THP and I like to have that available for those Services during Boot-Time.

This has been in place in the customers environment for several months and they have not reported any issues with this Unit file. I have also been using this same unit file on 3 different Indexers in my Repro Environment since November without any problems and I force Kill Splunk on a regular basis testing different aspects of Splunk and how it interacts with the Linux Kernel.

[mandrewjeski]

cheers mate, looks the business.