Hi all,
We recently ran into issues with our heavy forwarder being unable to connect to certain IPs in our Splunk Cloud environment.
07-22-2024 19:20:14.384 +0000 WARN AutoLoadBalancedConnectionStrategy [2042120 TcpOutEloop] - Cooked connection to ip=[[IP1]]:9997 timed out
07-22-2024 19:19:54.508 +0000 WARN AutoLoadBalancedConnectionStrategy [2042120 TcpOutEloop] - Cooked connection to ip=[[IP2]]:9997 timed out
07-22-2024 19:19:34.584 +0000 WARN AutoLoadBalancedConnectionStrategy [2042120 TcpOutEloop] - Cooked connection to ip=[[IP3]]:9997 timed out
This appears to be a pretty common error based on what I've seen in other community posts, and typically they are related to a firewall issue. I wanted to document that in our case, the issue was related to the IPs not being assigned to indexers on the Splunk Cloud instance.
According to Splunk support, "Usually, these DNS inputs (inputs1.companyname.splunkcloud.com, inputs2.........., inputs15.companyname.splunkcloud.com) resolve to the aforementioned IP tables defined, but these IPs should be then linked to the indexers which is not what we currently have present."
I assume this is an uncommon root cause, and wanted to put it out there as another troubleshooting option when investigating this issue. I hope it helps.
