Since last 2 days, we have started getting the following error when accessing any of our Splunk URL's.
This is effecting all our instances be it Search Head or Indexer. Nothing can be accessed. We do have a P1 case open with support and they are going through the logs, but I just wanted to put this out there in case anyone else has run into this before.
All our servers are high-spec Physical machines and resource usage is nominal. ulimit is set to a high number and lsof shows the # of open files are not even close to that limit. There were no recent changes/upgrades done to Splunk.
Splunk Version - 6.6.2
Host OS - CentOS 6.9
This is happening for all users irrespective of which browser they use. Cleared browser cache but that didn't help.
no specific errors under splunkd_ui_access.log. For each refresh, this is what we see:
X.X.X.X - - [03/Aug/2017:09:43:43.834 -0400] "GET /favicon.ico HTTP/1.1" 303 397 "https://Splunk_Hostname.x.y.local:8000/en-US/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.109 Safari/537.36" - - 0ms X.X.X.X - - [03/Aug/2017:09:43:43.836 -0400] "GET /en-US/favicon.ico HTTP/1.1" 502 154 "https://Splunk_Hostname.x.y.local:8000/en-US/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.109 Safari/537.36" - - 0ms
Is there anything else that I can check to isolate where the issue is?
I try first to downgrade 6.6.2 to 6.2.1. It still not working.
So, I downgraded to 6.5.5 and the interface cameback.
I did it at my risk. I did not wait for the Support.
Now I'll wait for the answer to try the new version.