Hey everyone,
recently I have been seeing some socket errors on my Splunk installation. I have a search head cluster and an indexer cluster.
When I load a dash board, some of the visual in it have red triagle at top right corner, which, on clicking shows message:
"Unknown error for peer . Search Results might be incomplete. If this occurs frequently, please check on the peer"
On indexer I see winsock 10053 errors in the splunkd.log at the time when I triggered the search through the dashboard:
06-18-2018 11:39:46.768 -0700 WARN HttpListener - Socket error from while accessing /services/streams/search: Winsock error 10053
At same time, there is a corresponding winsock 10053 error on the search head's splunkd.log:
06-18-2018 14:01:54.102 -0700 WARN HttpListener - Socket error from while accessing /en-US/splunkd/__raw/servicesNS/nobody/search/search/jobs/1529354406.43753_F791B43C-B641-4694-9E20-B5DE34C13007: Winsock error 10054
This is affecting all the search heads and indexers in their respective clusters, but these errors are intermittent and sometimes all the visuals in the dashboard load just fine without any errors. Any ideas how I can debug this?
