Splunk Enterprise

The splunkd daemon cannot be reached by splunkweb

splunkg
Explorer

Hello, since we upgraded to the version 10.4.3.0 we have encountered the problem that randomly after the login we hit the page with the following error message: 

Oops.

The splunkd daemon cannot be reached by splunkweb. Check that there are no blocked network ports or that splunkd is still running. Click here to return to Splunk homepage.

After a F5 Refresh the page loads again.. but that should not be the "solution".

After looking a bit trough the logs, the only error I found was this : 

2026-09-21 10:17:08,360 ERROR [6ab0e7e63b27b1af39010] decorators:357 - Splunkd daemon is not responding: ('Error connecting to /services/apps/local: The read operation timed out',)
Traceback (most recent call last):
File "D:\Splunk\Python-3.13\Lib\site-packages\splunk\rest\__init__.py", line 632, in simpleRequest
serverResponse, serverContent = h.request(uri, method, headers=headers, body=payload)
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

ERROR [6ab0e7e63b27b1af39010] __init__:659 - Socket error communicating with splunkd (error=The read operation timed out), path = /services/apps/local

Before the upgrade we did not had that problem. We did not change anything according to the firewall or ports.
Does anybody else encountered the same issue after upgrading ?

Labels (2)
0 Karma
1 Solution

livehybrid
SplunkTrust
SplunkTrust

Hi @splunkg 

I believe this might be a known issue, not a config problem on your end. Check out "Spike in 503 errors in the Splunk WebUI after upgrading to 10.0.3 (also affects 9.4.8/10.2.0)" - I think 10.4.3 has the same fault: 

Ultimately, recent releases added logic under [applicationsManagement] in server.conf that periodically pulls https://cdn.splunkbase.splunk.com/public/report/apps_dump.json to refresh the local apps cache used by /services/apps/local, even when allowInternetAccess = false. If your search head has no outbound internet access, the fetch hangs until it times out, and while it's hanging the /services/apps/local REST call that Splunk Web makes on every page load also stalls, throwing the "splunkd daemon cannot be reached" 503. The refresh/F5 "fix" works because the retry usually lands after the timeout has cleared.

Confirm it's the same fault by checking web_service.log around the timestamp for:

ERROR LocalAppsAdminHandler - Failed to fetch and parse apps dump; aborting splunkbase apps cache update

or

ERROR LocalAppsAdminHandler - Failed to parse apps dump uri from server.conf; aborting splunkbase apps cache update

There's a Splunk support KB specifically for this: "Error 503: Splunkd daemon cannot be reached by Splunk Web after upgrade to 9.4.8/10.x versions".

🌟 Did this answer help you? If so, please consider:

    • Adding karma to show it was useful
    • Marking it as the solution if it resolved your issue
    • Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing.

View solution in original post

livehybrid
SplunkTrust
SplunkTrust

Hi @splunkg 

I believe this might be a known issue, not a config problem on your end. Check out "Spike in 503 errors in the Splunk WebUI after upgrading to 10.0.3 (also affects 9.4.8/10.2.0)" - I think 10.4.3 has the same fault: 

Ultimately, recent releases added logic under [applicationsManagement] in server.conf that periodically pulls https://cdn.splunkbase.splunk.com/public/report/apps_dump.json to refresh the local apps cache used by /services/apps/local, even when allowInternetAccess = false. If your search head has no outbound internet access, the fetch hangs until it times out, and while it's hanging the /services/apps/local REST call that Splunk Web makes on every page load also stalls, throwing the "splunkd daemon cannot be reached" 503. The refresh/F5 "fix" works because the retry usually lands after the timeout has cleared.

Confirm it's the same fault by checking web_service.log around the timestamp for:

ERROR LocalAppsAdminHandler - Failed to fetch and parse apps dump; aborting splunkbase apps cache update

or

ERROR LocalAppsAdminHandler - Failed to parse apps dump uri from server.conf; aborting splunkbase apps cache update

There's a Splunk support KB specifically for this: "Error 503: Splunkd daemon cannot be reached by Splunk Web after upgrade to 9.4.8/10.x versions".

🌟 Did this answer help you? If so, please consider:

    • Adding karma to show it was useful
    • Marking it as the solution if it resolved your issue
    • Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing.

Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Tuning Edge Processor in Hybrid Mode: What Limits Each Data Path

When you run Edge Processor in hybrid mode, alongside a Splunk heavy forwarder on the same host, two questions ...

Best Practices: Splunk auto adjust pipeline queue

When you enable autoAdjustQueue in Splunk, maxSize should be understood as the queue size Splunk starts with ...

Request for Professional Development: Attending .conf26

Winning Over the Boss: Your Pass to .conf26 conf26 is going to be here before you know it. If don't already ...