Can someone please help with below error ?
Splunk forwarder is failing with below error.
● splunk.service - Systemd service file for Splunk, generated by 'splunk enable boot-start'
Loaded: loaded (/etc/systemd/system/splunk.service; enabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Fri 2020-02-21 14:11:39 PST; 785ms ago
Process: 30472 ExecStartPost=/bin/bash -c chown -R splunk:splunk /sys/fs/cgroup/memory/system.slice/%n (code=exited, status=0/SUCCESS)
Process: 30469 ExecStartPost=/bin/bash -c chown -R splunk:splunk /sys/fs/cgroup/cpu/system.slice/%n (code=exited, status=0/SUCCESS)
Process: 30468 ExecStart=/opt/splunk/splunkforwarder/bin/splunk _internal_launch_under_systemd (code=exited, status=1/FAILURE)
Main PID: 30468 (code=exited, status=1/FAILURE)
Feb 21 14:11:39 localhost systemd[1]: splunk.service: main process exited, code=exited, status=1/FAILURE
Feb 21 14:11:39 localhost systemd[1]: Unit splunk.service entered failed state.
Feb 21 14:11:39 localhost systemd[1]: splunk.service failed.
Feb 21 14:11:39 localhost systemd[1]: splunk.service holdoff time over, scheduling restart.
Feb 21 14:11:39 localhost systemd[1]: Stopped Systemd service file for Splunk, generated by 'splunk enable boot-start'.
Feb 21 14:11:39 localhost systemd[1]: start request repeated too quickly for splunk.service
Feb 21 14:11:39 localhost systemd[1]: Failed to start Systemd service file for Splunk, generated by 'splunk enable boot-start'.
Feb 21 14:11:39 localhost systemd[1]: Unit splunk.service entered failed state.
Feb 21 14:11:39 localhost systemd[1]: splunk.service failed.
Some people have reported issues with the systemd 'boot-start' scripts created by earlier versions of Splunk
In these cases running splunk disable boot-start
(removing the old config) and then re-enabling it with splunk enable boot-start
updates the script to the latest version and solves some problems.
Remember to set the boot-start options you need, eg -user splunk
and probably -systemd-managed 1
https://docs.splunk.com/Documentation/Splunk/8.0.2/Admin/ConfigureSplunktostartatboottime
Just wanted to toss in another possible resolution. If your system was ungracefully shutdown, resource exhaustion, virtualization host failure, etc. it's possible the splunkd.pid file wasn't cleaned up.
From what I have found, when splunkd attempts to start, it looks at this file and attempts various operations on the existing PIDs that are listed within. They are obviously not there and the service will fail to start. This will be made apparent when attempting to start splunk via the binary: SPLUNK_HOME/bin/splunk start
This file is located in SPLUNK_HOME/var/run/splunk/ and can be safely removed to correct. A new file will be created upon splunkd service starting successfully.
Some people have reported issues with the systemd 'boot-start' scripts created by earlier versions of Splunk
In these cases running splunk disable boot-start
(removing the old config) and then re-enabling it with splunk enable boot-start
updates the script to the latest version and solves some problems.
Remember to set the boot-start options you need, eg -user splunk
and probably -systemd-managed 1
https://docs.splunk.com/Documentation/Splunk/8.0.2/Admin/ConfigureSplunktostartatboottime
Just used this solution when upgrading from 8.0.3 to 8.2.6 Splunkd "disappeared" and only existed as a backup file named /etc/systemd/system/Splunkd.service_YYYY_MM_DD_HH_mm_SS
If my answer helped, please consider accepting and/or upvoting so that other memebers of the community can see it was useful.
No nickhillscpl. I have tried the way you suggested. But it didn't work.
Ok, so firstly to address you question subject:
"start request repeated too quickly for splunk.service"
That specific message is used when a service failes to (re)start a number of times in a short period of time (I think the default is 5 attempts within 10 seconds)
To try and work out what is happening, are you able to start splunk manually with $SPLUNK_HOME/bin/splunk start
?
If so, who are you starting Splunk as? root/splunk/or someone else?
yes, when i am able to start and stop using $SPLUNK_HOME/bin/splunk start using root and also using my user account.
However, if i use systemctl start splunk its failing
Ok, when you ran splunk enable boot-start
did you specify -user splunk
?
if yes, can you consider running:
sudo chown splunk:splunk /opt/splunk -R
This will give the splunk user & group ownership of the files in your Splunk installation.
If you are using systemctl to start Splunk as the splunk user it needs to own the contents of $SPLUNK_HOME
You can check if this is a problem by looking for files inside $SPLUNK_HOME which are owned by anyone other than splunk:splunk - if you have objects owned by root it may well be preventing splunk from launching correctly
yes, i did the same steps.
Please find below steps.
1) Killed all splunk process.
2) disabled boot start
3) Enabled boot start with -user splunk
4) verified all directories /opt/splunk has same splunk:splunk
5) started the splunk service using systemctl start splunk
but no luck
What happens if you manually start Splunk with the splunk user?
sudo su - splunk
followed by /opt/splunk/bin/splunk start
we have 1000's of servers and we are creating automation script to run the systemctl command on all servers to start, stop and check the status of all servers.
If it is working on other servers why the splunk is failing on this particular server is the question to me by my manager.
when i run manually sudo su - splunk followed by /opt/splunk/bin/splunk start
this works fine without any issue.
if it do /opt/splunk/bin/splunk status it shows running
but if i do systemctl status splunk it shows failed.
did you add the splunk group to the /etc/sudoers list with no password and provide the it permissions to start, stop, restart splunkd.service ?
EG:
%SplunkGroup ALL=NOPASSWD: /bin/systemctl start splunkd.service
Note: you will also need to restart the systemctl daemon.
you should then be able to start using: sudo systemctl start splunk
Hope this helps