Building for the Splunk Platform

Splunk Systemd Service

chrisitanmoleck
Path Finder

Hello,

Has anyone a working systemd script for Redhat/SUSE?

If I using the script from https://answers.splunk.com/answers/59662/is-there-a-systemd-unit-file-for-splunk.html
I get some error at the HTTP-Listener

10-17-2017 09:07:36.017 +0200 ERROR DispatchProcess - Failed to start the search process. 10-17-2017 09:07:36.032 +0200 ERROR SearchProcessRunner - Error reading from preforked process=0/25: Connection reset by peer 10-17-2017 09:07:36.123 +0200 WARN  Thread - HTTPDispatch: about to throw a ThreadException: pthread_create: Resource temporarily unavailable; 75 threads active 10-17-2017 09:07:36.123
+0200 ERROR HttpListener - Error spawning thread: HTTPDispatch: about to throw a ThreadException: pthread_create: Resource temporarily unavailable; 75 threads active 10-17-2017 09:07:45.273 +0200 ERROR SearchProcessRunner - preforked search=0/32 on process=0/31 caught exception.  completed_searches=0, process_started=1508224065.223881, search_started=1508224065.228171, search_ended=1508224065.273768, total_usage_time=0.046 10-17-2017 09:07:45.273 +0200 ERROR SearchProcessRunner - preforked process=0/31 died on exception: Main Thread: about to throw a ThreadException: pthread_create: Resource temporarily unavailable; 3 threads active 10-17-2017 09:07:50.688
+0200 WARN  ProcessTracker - executable=splunk-optimize failed to start reason='': Resource temporarily unavailable 10-17-2017 09:07:50.692
+0200 WARN  ProcessTracker - executable=splunk-optimize failed to start reason='': Resource temporarily unavailable 10-17-2017 09:07:50.693
+0200 WARN  ProcessTracker - executable=splunk-optimize failed to start reason='': Resource temporarily unavailable 10-17-2017 09:07:50.693
+0200 WARN  ProcessTracker - executable=splunk-optimize failed to start reason='': Resource temporarily unavailable
0 Karma
1 Solution

tonymata
Engager

I use this systemd script on my SLES 12 SP3 installation.

[Unit]
Description=Splunk Enterprise
After=network.target
Wants=network.target

[Service]
Type=forking
RemainAfterExit=False
User=<Enter_your_user_here>
Group=<Enter_your_group_here>
LimitNOFILE=65536
ExecStart=/opt/splunk/bin/splunk start --accept-license --answer-yes --no-prompt
ExecStop=/opt/splunk/bin/splunk stop
ExecReload=/opt/splunk/bin/splunk restart
PIDFile=/opt/splunk/var/run/splunk/splunkd.pid
TimeoutSec=600
TasksMax=infinity

[Install]
WantedBy=multi-user.target
Alias=splunk.service

Hopes this helps.

View solution in original post

Tags (1)

bandit
Motivator

Summary of the issue:
Splunk 6.0.0 - Splunk 7.2.1 defaults to using init.d when enabling boot start
Splunk 7.2.2 - Splunk 7.2.9 defaults to using systemd when enabling boot start
Splunk 7.3.0 - Splunk 8.x defaults to using init.d when enabling boot start

systemd defaults to prompting for root credentials upon stop/start/restart of Splunk

Here is a simple fix if you have encountered this issue and prefer to use the traditional init.d scripts vs systemd.

Splunk Enterprise/Heavy Forwarder example (note: replace the splunk user below with the account you run splunk as):

sudo /opt/splunk/bin/splunk disable boot-start
sudo /opt/splunk/bin/splunk enable boot-start -user splunk -systemd-managed 0

Splunk Universal Forwarder example (note: replace the splunk user below with the account you run splunk as):

sudo /opt/splunkforwarder/bin/splunk disable boot-start
sudo /opt/splunkforwarder/bin/splunk enable boot-start -user splunk -systemd-managed 0

tonymata
Engager

I use this systemd script on my SLES 12 SP3 installation.

[Unit]
Description=Splunk Enterprise
After=network.target
Wants=network.target

[Service]
Type=forking
RemainAfterExit=False
User=<Enter_your_user_here>
Group=<Enter_your_group_here>
LimitNOFILE=65536
ExecStart=/opt/splunk/bin/splunk start --accept-license --answer-yes --no-prompt
ExecStop=/opt/splunk/bin/splunk stop
ExecReload=/opt/splunk/bin/splunk restart
PIDFile=/opt/splunk/var/run/splunk/splunkd.pid
TimeoutSec=600
TasksMax=infinity

[Install]
WantedBy=multi-user.target
Alias=splunk.service

Hopes this helps.

Tags (1)

yuanliu
SplunkTrust
SplunkTrust

I came across this and tested with 8.1.2 successfully.  Meanwhile, as this is dated, Splunk now has official systemd support; see Run Splunk Enterprise as a systemd service.  Specifically, in Additional options for enable boot-start, a highlight panel states

Do not use the following properties. These properties can cause splunkd to fail on restart.
RemainAfterExit=yes
ExecStop

I didn't experience problem with  restart with ExecStop but it's probably prudent to just use the official guide.  Procedure is simple, just run 

[sudo] $SPLUNK_HOME/bin/splunk enable boot-start -systemd-managed 1 -user <username> -group <groupname>

 

0 Karma

graether
Path Finder

Thanks, the crucial part for me was 

TimeoutSec=600
TasksMax=infinity

For some reason it was not needed for release 7.2.5, but needed for 8.1 

0 Karma

chrisitanmoleck
Path Finder

Thank You tonymata.
Your script works very well.

0 Karma

chrisitanmoleck
Path Finder

Does any one has a idea or a usable systemd script for SLES?

0 Karma

dimrirahul
Explorer

Splunks latest version supports systemd file generation please look at https://docs.splunk.com/Documentation/Splunk/7.2.3/Admin/RunSplunkassystemdservice

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...