Getting Data In

systemd service fails during boot

denissotoacc
Path Finder

Hello,

We have an Splunk Enterprise version 8.0.5 configured in PRD, we have enabled boot-start with --systemd-managed 1 and specified the user splunk as the owner of the service. It keeps failing during boot with this error:

start request repeated too quickly for splunk.service

If I run "splunk start | restart | stop" it uses systemd to manage the process also (what is correct) and it works properly after boot. 

If I run "systemctl start splunk" after boot, the service starts ok. The problem is only during boot.

Server information:

NAME="Oracle Linux Server"
VERSION="7.9"

journalctl logs says:

Failed at step EXEC spawning /opt/splunk/bin/splunk: No such file or directory

-- Subject: Process /opt/splunk/bin/splunk could not be executed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- The process /opt/splunk/bin/splunk could not be executed and failed.
--
-- The error number returned by this process is 2.

 

How can we fix it?

Labels (5)
0 Karma
1 Solution

denissotoacc
Path Finder

[SOLVED]:

I had to append the following parameters to the [Service] tag in the unit file (splunk.service):

StartLimitInterval=30
StartLimitBurst=5
RestartSec=10

Now it's working correctly.

View solution in original post

0 Karma

maraman_splunk
Splunk Employee
Splunk Employee

Hi,

I find a bit unusual that your partition doesnt get mounted before splunk start but you can add a dependency on a mounted partition following something like https://unix.stackexchange.com/questions/246935/set-systemd-service-to-execute-after-fstab-mount

 

0 Karma

denissotoacc
Path Finder

[SOLVED]:

I had to append the following parameters to the [Service] tag in the unit file (splunk.service):

StartLimitInterval=30
StartLimitBurst=5
RestartSec=10

Now it's working correctly.

0 Karma

maraman_splunk
Splunk Employee
Splunk Employee

Hi,

do you have something occuring at start that could not have completed when splunk service is started nut still be needed by splunk ?

like a partition mount ? or a security feature ?

you could use systemd-analyze to understand your boot order

0 Karma

denissotoacc
Path Finder

Indeed, we have /opt/splunk as a partition mount. Should we add something to the unit file in order to wait for partition's initialization? How can we do that?

 

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...