Why does splunk service fails to start with error : ” Timed out waiting for splunkd to start” ?

[sdubey_splunk]

While trying to troubleshooting/fix the issue
1. No conclusive errors in log file as to why Splunkd fails to start.
2. We tried starting Splunk service in debug mode ie splunk start --debug : still we couldn't find the cause.
3. Checked Task manager( this on Window server) : there were no stale Windows servers
4. Rebooted the server still no luck.

Every time we try to start : we see ” Timed out waiting for splunkd to start” . How to fix?

[niketn]

@sdubey_splunk under Resource Monitor --> Network --> Listening Port is there any process other than splunkd already showing up against port 8000.

[sdubey_splunk · Accepted solution]

to restart splunk in debug mode. We found the issue was with log.cfg file. While try to enable debugging we saved log.cfg and
it got saved as log.cfg.txt. Once we changed the log.cfg.txt to log.cfg Splunk came up fine.

Root cause of issue of saving log.cfg as log.cfg.txt. Once we renamed log.cfg.txt to log.cfg Splunk started successfully.
Lesson learnt:
1. If possible try updating log.cfg manually better use CLI.
./splunk set log-level TailingProcessor -level DEBUG

1. Create and edit a $SPLUNK_HOME/etc/log-local.cfg