Splunk Systemd Service

[chrisitanmoleck]

Hello,

Has anyone a working systemd script for Redhat/SUSE?

If I using the script from https://answers.splunk.com/answers/59662/is-there-a-systemd-unit-file-for-splunk.html
I get some error at the HTTP-Listener

10-17-2017 09:07:36.017 +0200 ERROR DispatchProcess - Failed to start the search process. 10-17-2017 09:07:36.032 +0200 ERROR SearchProcessRunner - Error reading from preforked process=0/25: Connection reset by peer 10-17-2017 09:07:36.123 +0200 WARN  Thread - HTTPDispatch: about to throw a ThreadException: pthread_create: Resource temporarily unavailable; 75 threads active 10-17-2017 09:07:36.123
+0200 ERROR HttpListener - Error spawning thread: HTTPDispatch: about to throw a ThreadException: pthread_create: Resource temporarily unavailable; 75 threads active 10-17-2017 09:07:45.273 +0200 ERROR SearchProcessRunner - preforked search=0/32 on process=0/31 caught exception.  completed_searches=0, process_started=1508224065.223881, search_started=1508224065.228171, search_ended=1508224065.273768, total_usage_time=0.046 10-17-2017 09:07:45.273 +0200 ERROR SearchProcessRunner - preforked process=0/31 died on exception: Main Thread: about to throw a ThreadException: pthread_create: Resource temporarily unavailable; 3 threads active 10-17-2017 09:07:50.688
+0200 WARN  ProcessTracker - executable=splunk-optimize failed to start reason='': Resource temporarily unavailable 10-17-2017 09:07:50.692
+0200 WARN  ProcessTracker - executable=splunk-optimize failed to start reason='': Resource temporarily unavailable 10-17-2017 09:07:50.693
+0200 WARN  ProcessTracker - executable=splunk-optimize failed to start reason='': Resource temporarily unavailable 10-17-2017 09:07:50.693
+0200 WARN  ProcessTracker - executable=splunk-optimize failed to start reason='': Resource temporarily unavailable

[bandit]

Summary of the issue:
Splunk 6.0.0 - Splunk 7.2.1 defaults to using init.d when enabling boot start
Splunk 7.2.2 - Splunk 7.2.9 defaults to using systemd when enabling boot start
Splunk 7.3.0 - Splunk 8.x defaults to using init.d when enabling boot start

systemd defaults to prompting for root credentials upon stop/start/restart of Splunk

Here is a simple fix if you have encountered this issue and prefer to use the traditional init.d scripts vs systemd.

Splunk Enterprise/Heavy Forwarder example (note: replace the splunk user below with the account you run splunk as):

sudo /opt/splunk/bin/splunk disable boot-start
sudo /opt/splunk/bin/splunk enable boot-start -user splunk -systemd-managed 0

Splunk Universal Forwarder example (note: replace the splunk user below with the account you run splunk as):

sudo /opt/splunkforwarder/bin/splunk disable boot-start
sudo /opt/splunkforwarder/bin/splunk enable boot-start -user splunk -systemd-managed 0

[tonymata · Accepted solution]

I use this systemd script on my SLES 12 SP3 installation.

[Unit]
Description=Splunk Enterprise
After=network.target
Wants=network.target

[Service]
Type=forking
RemainAfterExit=False
User=<Enter_your_user_here>
Group=<Enter_your_group_here>
LimitNOFILE=65536
ExecStart=/opt/splunk/bin/splunk start --accept-license --answer-yes --no-prompt
ExecStop=/opt/splunk/bin/splunk stop
ExecReload=/opt/splunk/bin/splunk restart
PIDFile=/opt/splunk/var/run/splunk/splunkd.pid
TimeoutSec=600
TasksMax=infinity

[Install]
WantedBy=multi-user.target
Alias=splunk.service

Hopes this helps.

[yuanliu]

I came across this and tested with 8.1.2 successfully.  Meanwhile, as this is dated, Splunk now has official systemd support; see Run Splunk Enterprise as a systemd service.  Specifically, in Additional options for enable boot-start, a highlight panel states

Do not use the following properties. These properties can cause splunkd to fail on restart.
RemainAfterExit=yes
ExecStop

I didn't experience problem with  restart with ExecStop but it's probably prudent to just use the official guide.  Procedure is simple, just run 

[sudo] $SPLUNK_HOME/bin/splunk enable boot-start -systemd-managed 1 -user <username> -group <groupname>

[graether]

Thanks, the crucial part for me was 

TimeoutSec=600
TasksMax=infinity

For some reason it was not needed for release 7.2.5, but needed for 8.1 

[chrisitanmoleck]

Thank You tonymata.
Your script works very well.

[chrisitanmoleck]

Does any one has a idea or a usable systemd script for SLES?

[dimrirahul]

Splunks latest version supports systemd file generation please look at https://docs.splunk.com/Documentation/Splunk/7.2.3/Admin/RunSplunkassystemdservice