503 Service unavailable - Is my splunkd crashed ?

[chimbudp]

Could not able to access Splunkweb.
ERROR : 503 Service unavailable
The splunkd daemon cannot be reached by splunkweb. Check that there are no blocked network ports or that splunkd is still running.

Splunkd , Splunkweb - Services are running fine. VERIFIED

How can i troubleshoot this issue ?
Please help

[cartergray70543]

If both services are running, check whether Splunk Web can reach the 8089 management port. Review splunkd.log and web_service.log for errors, verify web.conf/server.conf, and make sure no firewall or proxy is blocking the connection._

[orezaie]

First, please check the Splunk log with:
grep -Ei 'LocalAppsAdminHandler|apps_dump|archived_apps_dump|services/apps/local|read operation timed out|Connect Timeout|splunkd daemon is not responding' \
/opt/splunk/var/log/splunk/splunkd.log | tail -100

If the output contains errors such as:
Could not fetch apps from apps dump
Could not fetch apps from archived apps dump
Error connecting: Connect Timeout

then add the following configuration to:
/opt/splunk/etc/system/local/server.conf
[applicationsManagement]
allowInternetAccess = false

Then restart Splunk:
/opt/splunk/bin/splunk restart

This should resolve the Splunk Web 503 issue caused by the Splunkbase app-dump connection timeouts.


Happy Splunk

[chskm]

The issue will raise at the time of splunk restart or else in some times due to network outage. Try logging into splunkweb for particular server. If you can able to login ten everything is working properly. Even you can check by loggin into indexers. ( cluster master to check the splunk is up or not)

[chimbudp]

Instance of splunk is running already and it enters into a CLOSE_WAIT state.
So , we need to kill the previously existing Splunk instance in the TASK MANAGER.

Its working now.

[krugger · Accepted solution]

There sure seems to be a problem. Usually get that when splunk is restarting and therefore is not ready to handle my web requests.

Run in the shell:
splunk status

Should say something like:
splunkd is running (PID: 16691).
splunk helpers are running (PIDs: 16692).
splunkweb is running (PID: 16991).

Have a look at /opt/splunk/var/log/splunk/splunkd.log.

[mehdihammami]

03-29-2021 04:00:35.158 +0100 WARN ProcessTracker - executable=splunk-optimize failed to start reason='': Média protégé en écriture. 

i have the same probleme in splunk on WINDOWS 10, when i had a look in splunk/var/log/splunk/splunkd.log he told splunk failed to start ( 03-29-2021 04:00:35.158 +0100 WARN ProcessTracker - executable=splunk-optimize failed to start reason='': Média protégé en écriture. ) 

[stefano_guidoba]

My first suggestion is to check splunk logs at $SPLUNK_HOME/var/log. Do they suggest anything?

[chskm]

The issue will raise at the time of splunk restart or else in some times due to network outage. Try logging into splunkweb for particular server. If you can able to login ten everything is working properly. Even you can check by loggin into indexers. ( cluster master to check the splunk is up or not)