topic Re: start request repeated too quickly for splunk.service in Getting Data In

start request repeated too quickly for splunk.service

venkateshparank — Wed, 30 Sep 2020 04:16:22 GMT

Can someone please help with below error ?
Splunk forwarder is failing with below error.

● splunk.service - Systemd service file for Splunk, generated by 'splunk enable boot-start'
Loaded: loaded (/etc/systemd/system/splunk.service; enabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Fri 2020-02-21 14:11:39 PST; 785ms ago
Process: 30472 ExecStartPost=/bin/bash -c chown -R splunk:splunk /sys/fs/cgroup/memory/system.slice/%n (code=exited, status=0/SUCCESS)
Process: 30469 ExecStartPost=/bin/bash -c chown -R splunk:splunk /sys/fs/cgroup/cpu/system.slice/%n (code=exited, status=0/SUCCESS)
Process: 30468 ExecStart=/opt/splunk/splunkforwarder/bin/splunk _internal_launch_under_systemd (code=exited, status=1/FAILURE)
Main PID: 30468 (code=exited, status=1/FAILURE)

Feb 21 14:11:39 localhost systemd[1]: splunk.service: main process exited, code=exited, status=1/FAILURE
Feb 21 14:11:39 localhost systemd[1]: Unit splunk.service entered failed state.
Feb 21 14:11:39 localhost systemd[1]: splunk.service failed.
Feb 21 14:11:39 localhost systemd[1]: splunk.service holdoff time over, scheduling restart.
Feb 21 14:11:39 localhost systemd[1]: Stopped Systemd service file for Splunk, generated by 'splunk enable boot-start'.
Feb 21 14:11:39 localhost systemd[1]: start request repeated too quickly for splunk.service
Feb 21 14:11:39 localhost systemd[1]: Failed to start Systemd service file for Splunk, generated by 'splunk enable boot-start'.
Feb 21 14:11:39 localhost systemd[1]: Unit splunk.service entered failed state.
Feb 21 14:11:39 localhost systemd[1]: splunk.service failed.

Re: start request repeated too quickly for splunk.service

nickhills — Sat, 22 Feb 2020 14:27:46 GMT

Some people have reported issues with the systemd 'boot-start' scripts created by earlier versions of Splunk
In these cases running splunk disable boot-start (removing the old config) and then re-enabling it with splunk enable boot-start updates the script to the latest version and solves some problems.

Remember to set the boot-start options you need, eg -user splunk and probably -systemd-managed 1
https://docs.splunk.com/Documentation/Splunk/8.0.2/Admin/ConfigureSplunktostartatboottime

Re: start request repeated too quickly for splunk.service

nickhills — Wed, 26 Feb 2020 21:54:37 GMT

If my answer helped, please consider accepting and/or upvoting so that other memebers of the community can see it was useful.

Re: start request repeated too quickly for splunk.service

venkateshparank — Wed, 26 Feb 2020 22:00:07 GMT

No nickhillscpl. I have tried the way you suggested. But it didn't work.

Re: start request repeated too quickly for splunk.service

nickhills — Wed, 26 Feb 2020 22:37:18 GMT

Ok, so firstly to address you question subject:
"start request repeated too quickly for splunk.service"

That specific message is used when a service failes to (re)start a number of times in a short period of time (I think the default is 5 attempts within 10 seconds)

To try and work out what is happening, are you able to start splunk manually with $SPLUNK_HOME/bin/splunk start?
If so, who are you starting Splunk as? root/splunk/or someone else?

Re: start request repeated too quickly for splunk.service

venkateshparank — Wed, 26 Feb 2020 22:40:08 GMT

yes, when i am able to start and stop using $SPLUNK_HOME/bin/splunk start using root and also using my user account.
However, if i use systemctl start splunk its failing

Re: start request repeated too quickly for splunk.service

nickhills — Wed, 26 Feb 2020 22:48:53 GMT

Ok, when you ran splunk enable boot-start did you specify -user splunk?

if yes, can you consider running:
sudo chown splunk:splunk /opt/splunk -R

This will give the splunk user & group ownership of the files in your Splunk installation.
If you are using systemctl to start Splunk as the splunk user it needs to own the contents of $SPLUNK_HOME

You can check if this is a problem by looking for files inside $SPLUNK_HOME which are owned by anyone other than splunk:splunk - if you have objects owned by root it may well be preventing splunk from launching correctly

Re: start request repeated too quickly for splunk.service

venkateshparank — Wed, 26 Feb 2020 23:01:14 GMT

yes, i did the same steps.
Please find below steps.

1) Killed all splunk process.
2) disabled boot start
3) Enabled boot start with -user splunk
4) verified all directories /opt/splunk has same splunk:splunk
5) started the splunk service using systemctl start splunk

but no luck

Re: start request repeated too quickly for splunk.service

nickhills — Wed, 26 Feb 2020 23:06:00 GMT

What happens if you manually start Splunk with the splunk user?

sudo su - splunk followed by /opt/splunk/bin/splunk start

Re: start request repeated too quickly for splunk.service

venkateshparank — Wed, 26 Feb 2020 23:27:00 GMT

we have 1000's of servers and we are creating automation script to run the systemctl command on all servers to start, stop and check the status of all servers.
If it is working on other servers why the splunk is failing on this particular server is the question to me by my manager.

Re: start request repeated too quickly for splunk.service

venkateshparank — Wed, 26 Feb 2020 23:33:40 GMT

when i run manually sudo su - splunk followed by /opt/splunk/bin/splunk start
this works fine without any issue.
if it do /opt/splunk/bin/splunk status it shows running
but if i do systemctl status splunk it shows failed.

Re: start request repeated too quickly for splunk.service

anmolpatel — Thu, 27 Feb 2020 00:09:22 GMT

did you add the splunk group to the /etc/sudoers list with no password and provide the it permissions to start, stop, restart splunkd.service ?

EG:
%SplunkGroup ALL=NOPASSWD: /bin/systemctl start splunkd.service

Note: you will also need to restart the systemctl daemon.

you should then be able to start using: sudo systemctl start splunk

Hope this helps

Re: start request repeated too quickly for splunk.service

Sir_Redan — Thu, 02 Jun 2022 12:28:03 GMT

Just used this solution when upgrading from 8.0.3 to 8.2.6 Splunkd "disappeared" and only existed as a backup file named /etc/systemd/system/Splunkd.service_YYYY_MM_DD_HH_mm_SS

Re: start request repeated too quickly for splunk.service

GfussVMW — Fri, 17 Nov 2023 18:58:57 GMT

Just wanted to toss in another possible resolution. If your system was ungracefully shutdown, resource exhaustion, virtualization host failure, etc. it's possible the splunkd.pid file wasn't cleaned up.

From what I have found, when splunkd attempts to start, it looks at this file and attempts various operations on the existing PIDs that are listed within. They are obviously not there and the service will fail to start. This will be made apparent when attempting to start splunk via the binary: SPLUNK_HOME/bin/splunk start

This file is located in SPLUNK_HOME/var/run/splunk/ and can be safely removed to correct. A new file will be created upon splunkd service starting successfully.