For the past few days, after upgrading the infrastructure from 7.3.2 to the latest GA (8.0.5), I'm having problems when running ad-hoc searches on an SHC. To give you more context about the Splunk infrastructure I'm talking about, I've described it at the end of the post.
Following is the problem I'm facing:
When I connect to the SHC using the VIP and I run whatever search, the system raises the following error after 5-10 seconds. I couldn't find any relevant information by looking at the logs.
When I connect directly to any of the Search Heads and I run the same search, it runs smoothly without any problem.
I found the following Known Issues (SPL-192057, SPL-188608) that seem to match this behavior. These are pretty recent though, but I can't find which Splunk versions are affected.
Did anyone face this before? What do you think I should do?
3 Search Heads
These SH are in a Search Head Cluster (SHC) configured to distribute the searches on both Indexers
Load balancer in front of the SHC
2 Heavy Forwarders + multiple Universal Forwarders