We are currently experiencing an issue in our 9.0.2 environment where our syslog UFs are unable to connect to our indexers. When we take a look at the splunkd.log on our syslog servers we see:
WARN AutoLoadBalancedConnectionStrategy [3438113 TcpOutEloop] - Cooked connection to ip=xxx.xxx.xxx.xxx:9997 timed out
These servers are in the same VRF so there is no firewall in-between, we have useACK and autoBatch set to false for the 9.x workaround, and the indexers are receiving all data from our non-syslog UFs. These syslog servers had been working just fine up until a day or two ago. If anyone has additional t/s suggestions that'd be much appreciated
Hi
1st you must check what has changed on that time after those are stopped to work. Probably some changes e.g. those UFs or servers have restarted, or some other network OS level change has deployed. Quite often these issues have arise after restart even actual changes e.g. in conf files has done much much earlier (but restart takes those into effect).
Message said that there is no s2s connection between UF and Idx. You can try it with e.g. curl and look at same time with tcpdump what is happening on network level. If you haven't use those tools earlier, I propose that try to find someone familiar with those to help you.
r. Ismo