I am having an issue , we have 3 search heads in cluster and are currently handle by a load balancer. some times my Web GUI is too slow its not even loading the search bar is not coming up , its getting hang and we can only see loading screen. I am unable to find out whats the issue , is it the load balance or networking issue not sure , If any one can guide me on it will be helpfull.
Check with your team that manages the load balancer.
Make sure you have the required ports for Splunk (web, management port, forwarding) enabled on the load balancer as well as any required healthcheck mechanisms (internal services, APIs, healthcheck ports, etc) required by the load balancer.
The issue happens again today and i checked the internal logs my index is full with these watchdog logs , is it impacting the timeout issue ?
Watchdog - No response received from IMonitoredThread=0x7fa0e99fdba0 within 8000 ms.
For me, whenever our indexers are down, we face this issue on search heads, Check if indexers are good, check _internal log for below error on search heads. Try doing SH rolling restart for a temporary fix.
"Cannot determine a latest common bundle, search may be blocked"
Having only three heads in your cluster is tricky. It doesn't allow for one head to be down.
If at all possible I would suggest adding another head so that you can take one head out to do maintenance etc.
Again look through the _internal logs. Is the captain reporting any issues?
Can you look at the SH Cluster status? https://docs.splunk.com/Documentation/Splunk/7.3.4/DistSearch/ViewSHCstatus
yeah i checked the shcluster status and its showing all 3 are up only and in splunkd , i saw some error messages but i dont think those are relevent because i am not seeing any common patern of error messages when the issue is happening.
checked the internal logs but not clue and the weired thing is when this happens we tried to open indiviual search heads and their we are not seeing this issue , seems its just coming on the load balancer.