Unable to forward to indexer - cooked connection timed out

[wu_weidong]

Hi,

I'm trying to forward data into my Splunk indexer, but when I do a "./splunk list forward-server", it shows up under "Configured but inactive forwards". When I checked "splunkd.log", I see

AutoLoadBalancedConnectionStrategy [1676 TcpOutEloop] - Cooked connection to ip=<indexer_ip>:9997 timed out  <-- 5 times, every 20 seconds

TcpOutputProc [1675 parsing] - The TCP output processer has paused the data flow. Forwarding to host_dest=<indexer_ip> inside output group default-autolb-group from host_src=<forwarder hostname> has been blocked for blocked_seconds=8100. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

My forwarder is version 8.2.3 (on Ubuntu 20.04), my indexer is version 7.1.4 (on RHEL 7.4).

I'm using a non-root account on my forwarder, and every time I run splunk or access the monitored folders, I need to run the commands using "sudo".

I have tried to telnet from the forwarder to the indexer. On port 8089, I immediately get a "connection refused" message. On port 9997, there is no immediate response, the prompt just seems to wait for a long time at "Trying <indexer IP>...". I don't expect the telnet connection to be successful as my forwarder is behind a firewall that is very strict about which ports are allowed. But port 9997 (as destination) should be allowed on the network-level firewall.

I also tried to check my "ufw". But I don't think it's running.

> sudo ufw status
Status: inactive

> sudo systemctl status ufw.service
...
Active: active (exited) since Monday 2021-12-06; 2h 59min ago
...

> sudo ufw show listening
tcp:
   22 <forwarder IP>
   8089 * 
udp:
   ...

On the indexer, I have "/opt/splunk/etc/system/local/inputs.conf":

[default]
host = <indexer hostname>

[splunktcp://9997]
disabled = 0

What is wrong with my setup?