Getting Data In

start request repeated too quickly for splunk.service

venkateshparank
Path Finder

Can someone please help with below error ?
Splunk forwarder is failing with below error.

● splunk.service - Systemd service file for Splunk, generated by 'splunk enable boot-start'
Loaded: loaded (/etc/systemd/system/splunk.service; enabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Fri 2020-02-21 14:11:39 PST; 785ms ago
Process: 30472 ExecStartPost=/bin/bash -c chown -R splunk:splunk /sys/fs/cgroup/memory/system.slice/%n (code=exited, status=0/SUCCESS)
Process: 30469 ExecStartPost=/bin/bash -c chown -R splunk:splunk /sys/fs/cgroup/cpu/system.slice/%n (code=exited, status=0/SUCCESS)
Process: 30468 ExecStart=/opt/splunk/splunkforwarder/bin/splunk _internal_launch_under_systemd (code=exited, status=1/FAILURE)
Main PID: 30468 (code=exited, status=1/FAILURE)

Feb 21 14:11:39 localhost systemd[1]: splunk.service: main process exited, code=exited, status=1/FAILURE
Feb 21 14:11:39 localhost systemd[1]: Unit splunk.service entered failed state.
Feb 21 14:11:39 localhost systemd[1]: splunk.service failed.
Feb 21 14:11:39 localhost systemd[1]: splunk.service holdoff time over, scheduling restart.
Feb 21 14:11:39 localhost systemd[1]: Stopped Systemd service file for Splunk, generated by 'splunk enable boot-start'.
Feb 21 14:11:39 localhost systemd[1]: start request repeated too quickly for splunk.service
Feb 21 14:11:39 localhost systemd[1]: Failed to start Systemd service file for Splunk, generated by 'splunk enable boot-start'.
Feb 21 14:11:39 localhost systemd[1]: Unit splunk.service entered failed state.
Feb 21 14:11:39 localhost systemd[1]: splunk.service failed.

0 Karma
1 Solution

nickhills
Ultra Champion

Some people have reported issues with the systemd 'boot-start' scripts created by earlier versions of Splunk
In these cases running splunk disable boot-start (removing the old config) and then re-enabling it with splunk enable boot-start updates the script to the latest version and solves some problems.

Remember to set the boot-start options you need, eg -user splunk and probably -systemd-managed 1
https://docs.splunk.com/Documentation/Splunk/8.0.2/Admin/ConfigureSplunktostartatboottime

If my comment helps, please give it a thumbs up!

View solution in original post

GfussVMW
Engager

Just wanted to toss in another possible resolution.  If your system was ungracefully shutdown, resource exhaustion, virtualization host failure, etc. it's possible the splunkd.pid file wasn't cleaned up. 

From what I have found, when splunkd attempts to start, it looks at this file and attempts various operations on the existing PIDs that are listed within.  They are obviously not there and the service will fail to start.  This will be made apparent when attempting to start splunk via the binary:  SPLUNK_HOME/bin/splunk start

This file is located in SPLUNK_HOME/var/run/splunk/ and can be safely removed to correct.  A new file will be created upon splunkd service starting successfully. 

 

nickhills
Ultra Champion

Some people have reported issues with the systemd 'boot-start' scripts created by earlier versions of Splunk
In these cases running splunk disable boot-start (removing the old config) and then re-enabling it with splunk enable boot-start updates the script to the latest version and solves some problems.

Remember to set the boot-start options you need, eg -user splunk and probably -systemd-managed 1
https://docs.splunk.com/Documentation/Splunk/8.0.2/Admin/ConfigureSplunktostartatboottime

If my comment helps, please give it a thumbs up!

Sir_Redan
Explorer

Just used this solution when upgrading from 8.0.3 to 8.2.6 Splunkd "disappeared" and only existed as a backup file named /etc/systemd/system/Splunkd.service_YYYY_MM_DD_HH_mm_SS

0 Karma

nickhills
Ultra Champion

If my answer helped, please consider accepting and/or upvoting so that other memebers of the community can see it was useful.

If my comment helps, please give it a thumbs up!
0 Karma

venkateshparank
Path Finder

No nickhillscpl. I have tried the way you suggested. But it didn't work.

0 Karma

nickhills
Ultra Champion

Ok, so firstly to address you question subject:
"start request repeated too quickly for splunk.service"

That specific message is used when a service failes to (re)start a number of times in a short period of time (I think the default is 5 attempts within 10 seconds)

To try and work out what is happening, are you able to start splunk manually with $SPLUNK_HOME/bin/splunk start?
If so, who are you starting Splunk as? root/splunk/or someone else?

If my comment helps, please give it a thumbs up!
0 Karma

venkateshparank
Path Finder

yes, when i am able to start and stop using $SPLUNK_HOME/bin/splunk start using root and also using my user account.
However, if i use systemctl start splunk its failing

0 Karma

nickhills
Ultra Champion

Ok, when you ran splunk enable boot-start did you specify -user splunk?

if yes, can you consider running:
sudo chown splunk:splunk /opt/splunk -R

This will give the splunk user & group ownership of the files in your Splunk installation.
If you are using systemctl to start Splunk as the splunk user it needs to own the contents of $SPLUNK_HOME

You can check if this is a problem by looking for files inside $SPLUNK_HOME which are owned by anyone other than splunk:splunk - if you have objects owned by root it may well be preventing splunk from launching correctly

If my comment helps, please give it a thumbs up!
0 Karma

venkateshparank
Path Finder

yes, i did the same steps.
Please find below steps.

1) Killed all splunk process.
2) disabled boot start
3) Enabled boot start with -user splunk
4) verified all directories /opt/splunk has same splunk:splunk
5) started the splunk service using systemctl start splunk

but no luck

0 Karma

nickhills
Ultra Champion

What happens if you manually start Splunk with the splunk user?

sudo su - splunk followed by /opt/splunk/bin/splunk start

If my comment helps, please give it a thumbs up!
0 Karma

venkateshparank
Path Finder

we have 1000's of servers and we are creating automation script to run the systemctl command on all servers to start, stop and check the status of all servers.
If it is working on other servers why the splunk is failing on this particular server is the question to me by my manager.

0 Karma

venkateshparank
Path Finder

when i run manually sudo su - splunk followed by /opt/splunk/bin/splunk start
this works fine without any issue.
if it do /opt/splunk/bin/splunk status it shows running
but if i do systemctl status splunk it shows failed.

0 Karma

anmolpatel
Builder

did you add the splunk group to the /etc/sudoers list with no password and provide the it permissions to start, stop, restart splunkd.service ?

EG:
%SplunkGroup ALL=NOPASSWD: /bin/systemctl start splunkd.service

Note: you will also need to restart the systemctl daemon.

you should then be able to start using: sudo systemctl start splunk

Hope this helps

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...