Splunk Enterprise

The splunkd daemon cannot be reached by splunkweb

splunkg
Explorer

Hello, since we upgraded to the version 10.4.3.0 we have encountered the problem that randomly after the login we hit the page with the following error message: 

Oops.

The splunkd daemon cannot be reached by splunkweb. Check that there are no blocked network ports or that splunkd is still running. Click here to return to Splunk homepage.

After a F5 Refresh the page loads again.. but that should not be the "solution".

After looking a bit trough the logs, the only error I found was this : 

2026-09-21 10:17:08,360 ERROR [6ab0e7e63b27b1af39010] decorators:357 - Splunkd daemon is not responding: ('Error connecting to /services/apps/local: The read operation timed out',)
Traceback (most recent call last):
File "D:\Splunk\Python-3.13\Lib\site-packages\splunk\rest\__init__.py", line 632, in simpleRequest
serverResponse, serverContent = h.request(uri, method, headers=headers, body=payload)
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

ERROR [6ab0e7e63b27b1af39010] __init__:659 - Socket error communicating with splunkd (error=The read operation timed out), path = /services/apps/local

Before the upgrade we did not had that problem. We did not change anything according to the firewall or ports.
Does anybody else encountered the same issue after upgrading ?

Labels (2)
0 Karma
1 Solution

livehybrid
SplunkTrust
SplunkTrust

Hi @splunkg 

I believe this might be a known issue, not a config problem on your end. Check out "Spike in 503 errors in the Splunk WebUI after upgrading to 10.0.3 (also affects 9.4.8/10.2.0)" - I think 10.4.3 has the same fault: 

Ultimately, recent releases added logic under [applicationsManagement] in server.conf that periodically pulls https://cdn.splunkbase.splunk.com/public/report/apps_dump.json to refresh the local apps cache used by /services/apps/local, even when allowInternetAccess = false. If your search head has no outbound internet access, the fetch hangs until it times out, and while it's hanging the /services/apps/local REST call that Splunk Web makes on every page load also stalls, throwing the "splunkd daemon cannot be reached" 503. The refresh/F5 "fix" works because the retry usually lands after the timeout has cleared.

Confirm it's the same fault by checking web_service.log around the timestamp for:

ERROR LocalAppsAdminHandler - Failed to fetch and parse apps dump; aborting splunkbase apps cache update

or

ERROR LocalAppsAdminHandler - Failed to parse apps dump uri from server.conf; aborting splunkbase apps cache update

There's a Splunk support KB specifically for this: "Error 503: Splunkd daemon cannot be reached by Splunk Web after upgrade to 9.4.8/10.x versions".

🌟 Did this answer help you? If so, please consider:

    • Adding karma to show it was useful
    • Marking it as the solution if it resolved your issue
    • Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing.

View solution in original post

livehybrid
SplunkTrust
SplunkTrust

Hi @splunkg 

I believe this might be a known issue, not a config problem on your end. Check out "Spike in 503 errors in the Splunk WebUI after upgrading to 10.0.3 (also affects 9.4.8/10.2.0)" - I think 10.4.3 has the same fault: 

Ultimately, recent releases added logic under [applicationsManagement] in server.conf that periodically pulls https://cdn.splunkbase.splunk.com/public/report/apps_dump.json to refresh the local apps cache used by /services/apps/local, even when allowInternetAccess = false. If your search head has no outbound internet access, the fetch hangs until it times out, and while it's hanging the /services/apps/local REST call that Splunk Web makes on every page load also stalls, throwing the "splunkd daemon cannot be reached" 503. The refresh/F5 "fix" works because the retry usually lands after the timeout has cleared.

Confirm it's the same fault by checking web_service.log around the timestamp for:

ERROR LocalAppsAdminHandler - Failed to fetch and parse apps dump; aborting splunkbase apps cache update

or

ERROR LocalAppsAdminHandler - Failed to parse apps dump uri from server.conf; aborting splunkbase apps cache update

There's a Splunk support KB specifically for this: "Error 503: Splunkd daemon cannot be reached by Splunk Web after upgrade to 9.4.8/10.x versions".

🌟 Did this answer help you? If so, please consider:

    • Adding karma to show it was useful
    • Marking it as the solution if it resolved your issue
    • Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing.

Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Meet the .conf26 AI Agent Buildathon Winners

At the .conf26 AI Agent Buildathon, Splunk users turned their operational expertise into working AI agents ...

Trusted Autonomy in the SOC: A Maturity Journey

AI is compressing attacker timelines and pushing SOC teams beyond human capacity thresholds. The next SOC ...

Supercharging Windows Security Detection Performance: Introducing Hybrid Field ...

Windows event logs—from Security auditing and Sysmon to PowerShell script blocks—form the operational backbone ...