Developing for Splunk Platform

Splunk Systemd Service

chrisitanmoleck
Path Finder

Hello,

Has anyone a working systemd script for Redhat/SUSE?

If I using the script from https://answers.splunk.com/answers/59662/is-there-a-systemd-unit-file-for-splunk.html
I get some error at the HTTP-Listener

10-17-2017 09:07:36.017 +0200 ERROR DispatchProcess - Failed to start the search process. 10-17-2017 09:07:36.032 +0200 ERROR SearchProcessRunner - Error reading from preforked process=0/25: Connection reset by peer 10-17-2017 09:07:36.123 +0200 WARN  Thread - HTTPDispatch: about to throw a ThreadException: pthread_create: Resource temporarily unavailable; 75 threads active 10-17-2017 09:07:36.123
+0200 ERROR HttpListener - Error spawning thread: HTTPDispatch: about to throw a ThreadException: pthread_create: Resource temporarily unavailable; 75 threads active 10-17-2017 09:07:45.273 +0200 ERROR SearchProcessRunner - preforked search=0/32 on process=0/31 caught exception.  completed_searches=0, process_started=1508224065.223881, search_started=1508224065.228171, search_ended=1508224065.273768, total_usage_time=0.046 10-17-2017 09:07:45.273 +0200 ERROR SearchProcessRunner - preforked process=0/31 died on exception: Main Thread: about to throw a ThreadException: pthread_create: Resource temporarily unavailable; 3 threads active 10-17-2017 09:07:50.688
+0200 WARN  ProcessTracker - executable=splunk-optimize failed to start reason='': Resource temporarily unavailable 10-17-2017 09:07:50.692
+0200 WARN  ProcessTracker - executable=splunk-optimize failed to start reason='': Resource temporarily unavailable 10-17-2017 09:07:50.693
+0200 WARN  ProcessTracker - executable=splunk-optimize failed to start reason='': Resource temporarily unavailable 10-17-2017 09:07:50.693
+0200 WARN  ProcessTracker - executable=splunk-optimize failed to start reason='': Resource temporarily unavailable
0 Karma
1 Solution

tonymata
Engager

I use this systemd script on my SLES 12 SP3 installation.

[Unit]
Description=Splunk Enterprise
After=network.target
Wants=network.target

[Service]
Type=forking
RemainAfterExit=False
User=<Enter_your_user_here>
Group=<Enter_your_group_here>
LimitNOFILE=65536
ExecStart=/opt/splunk/bin/splunk start --accept-license --answer-yes --no-prompt
ExecStop=/opt/splunk/bin/splunk stop
ExecReload=/opt/splunk/bin/splunk restart
PIDFile=/opt/splunk/var/run/splunk/splunkd.pid
TimeoutSec=600
TasksMax=infinity

[Install]
WantedBy=multi-user.target
Alias=splunk.service

Hopes this helps.

View solution in original post

Tags (1)

bandit
Motivator

Summary of the issue:
Splunk 6.0.0 - Splunk 7.2.1 defaults to using init.d when enabling boot start
Splunk 7.2.2 - Splunk 7.2.9 defaults to using systemd when enabling boot start
Splunk 7.3.0 - Splunk 8.x defaults to using init.d when enabling boot start

systemd defaults to prompting for root credentials upon stop/start/restart of Splunk

Here is a simple fix if you have encountered this issue and prefer to use the traditional init.d scripts vs systemd.

Splunk Enterprise/Heavy Forwarder example (note: replace the splunk user below with the account you run splunk as):

sudo /opt/splunk/bin/splunk disable boot-start
sudo /opt/splunk/bin/splunk enable boot-start -user splunk -systemd-managed 0

Splunk Universal Forwarder example (note: replace the splunk user below with the account you run splunk as):

sudo /opt/splunkforwarder/bin/splunk disable boot-start
sudo /opt/splunkforwarder/bin/splunk enable boot-start -user splunk -systemd-managed 0

tonymata
Engager

I use this systemd script on my SLES 12 SP3 installation.

[Unit]
Description=Splunk Enterprise
After=network.target
Wants=network.target

[Service]
Type=forking
RemainAfterExit=False
User=<Enter_your_user_here>
Group=<Enter_your_group_here>
LimitNOFILE=65536
ExecStart=/opt/splunk/bin/splunk start --accept-license --answer-yes --no-prompt
ExecStop=/opt/splunk/bin/splunk stop
ExecReload=/opt/splunk/bin/splunk restart
PIDFile=/opt/splunk/var/run/splunk/splunkd.pid
TimeoutSec=600
TasksMax=infinity

[Install]
WantedBy=multi-user.target
Alias=splunk.service

Hopes this helps.

Tags (1)

yuanliu
SplunkTrust
SplunkTrust

I came across this and tested with 8.1.2 successfully.  Meanwhile, as this is dated, Splunk now has official systemd support; see Run Splunk Enterprise as a systemd service.  Specifically, in Additional options for enable boot-start, a highlight panel states

Do not use the following properties. These properties can cause splunkd to fail on restart.
RemainAfterExit=yes
ExecStop

I didn't experience problem with  restart with ExecStop but it's probably prudent to just use the official guide.  Procedure is simple, just run 

[sudo] $SPLUNK_HOME/bin/splunk enable boot-start -systemd-managed 1 -user <username> -group <groupname>

 

0 Karma

graether
Path Finder

Thanks, the crucial part for me was 

TimeoutSec=600
TasksMax=infinity

For some reason it was not needed for release 7.2.5, but needed for 8.1 

0 Karma

chrisitanmoleck
Path Finder

Thank You tonymata.
Your script works very well.

0 Karma

chrisitanmoleck
Path Finder

Does any one has a idea or a usable systemd script for SLES?

0 Karma

dimrirahul
Explorer

Splunks latest version supports systemd file generation please look at https://docs.splunk.com/Documentation/Splunk/7.2.3/Admin/RunSplunkassystemdservice

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...