Good afternoon!
This week we upgraded a Splunk deployment from 9.4.x to 10.0.3, and whilst everything seemingly went well, we came to work the next day to a number of upset users who kept getting random 503 errors and the Oops page:
I was able to replicate this and found that approximately every 5 minutes certain pages in the Splunk UI gave this frustating error! Following further analysis we found that 30 seconds after the 503 error we saw the following error in the _internal logs:
Splunkd daemon is not responding: ('Error connecting to /services/apps/local: The read operation timed out')
After checking directly to https://splunkserver:8089/services/apps/local we found that most of the time it returned in 10-30milliseconds but would periodically take upto 180 seconds to respond.
I later found errors in _internal for the following web address: https://cdn.splunkbase.splunk.com/public/report/apps_dump.json - This environment does not have internet access and has allowInternetAccess=false in server.conf.
Following a chat with some other users in the Community on Slack we found others have had the same issue with different versions - we found that a setting was added in the latest releases (9.4.8/10.0.3/10.2.0) [see https://github.com/livehybrid/splunk-spec-files/blob/master/server.conf#L255] in server.conf/[applicationsManagement]/splunkbaseAppsDumpUrl which contains that SplunkBase URL.
These settings were not in previous 9.4/10.0 releases and I couldnt find any reference to it in the Release Notes either, however settings this to a blank value solved the problem for us.
# server.conf
[applicationsManagement]
splunkbaseAppsDumpUrl=
archivedSplunkbaseAppsDumpUrl=Ultimately we believe this to be a bug which was introduced in the latest releases and have raised with Support to see if it gets added to the Known Issue for 10.0.3 (https://help.splunk.com/en/splunk-enterprise/release-notes-and-updates/release-notes/10.0/known-issu...)
Hopefully if you come across this issue you can find this fix! 🙂
🌟 Did this answer help you? If so, please consider:
- Adding karma to show it was useful
- Marking it as the solution if it resolved your issue
- Commenting if you need any clarification
Your feedback encourages the volunteers in this community to continue contributing
