systemd service fails during boot

[denissotoacc]

Hello,

We have an Splunk Enterprise version 8.0.5 configured in PRD, we have enabled boot-start with --systemd-managed 1 and specified the user splunk as the owner of the service. It keeps failing during boot with this error:

start request repeated too quickly for splunk.service

If I run "splunk start | restart | stop" it uses systemd to manage the process also (what is correct) and it works properly after boot. 

If I run "systemctl start splunk" after boot, the service starts ok. The problem is only during boot.

Server information:

NAME="Oracle Linux Server"
VERSION="7.9"

journalctl logs says:

Failed at step EXEC spawning /opt/splunk/bin/splunk: No such file or directory

-- Subject: Process /opt/splunk/bin/splunk could not be executed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- The process /opt/splunk/bin/splunk could not be executed and failed.
--
-- The error number returned by this process is 2.

How can we fix it?

[maraman_splunk]

Hi,

I find a bit unusual that your partition doesnt get mounted before splunk start but you can add a dependency on a mounted partition following something like https://unix.stackexchange.com/questions/246935/set-systemd-service-to-execute-after-fstab-mount

[denissotoacc · Accepted solution]

[SOLVED]:

I had to append the following parameters to the [Service] tag in the unit file (splunk.service):

StartLimitInterval=30
StartLimitBurst=5
RestartSec=10

Now it's working correctly.

[maraman_splunk]

Hi,

do you have something occuring at start that could not have completed when splunk service is started nut still be needed by splunk ?

like a partition mount ? or a security feature ?

you could use systemd-analyze to understand your boot order

[denissotoacc]

Indeed, we have /opt/splunk as a partition mount. Should we add something to the unit file in order to wait for partition's initialization? How can we do that?