FwdDataReceiverThread error | Read Timeout Timed out after 600 seconds

[GaetanVP]

Hello Splunkers,

I the following error on my Splunk HF which is listening to incoming data from F5 network appliance.

01-25-2023 08:06:56.794 +0000 ERROR TcpInputProc [2612981 FwdDataReceiverThread] - Error encountered for connection from src=<internal_ip_f5>:59697. Read Timeout Timed out after 600 seconds.

I am wondering what the number after the F5 IP is... I specified a unique port for the forwarding of data between f5 and the HF so I do not understand why I have number like 59697 (and many others).

More generally I do not know how to troubleshoot this...

Thanks for your help,

GaetanVP

[scelikok]

Hi @GaetanVP,

Yes, they work on the same machine. Syslog server writes logs to the filesystem, on the same machine HF or UF will use monitor input to ingest the data.

[scelikok · Accepted solution]

Hi @GaetanVP,

Let's assume you configured HF to listen TCP 5140 port and configured F5 to send to TCP 5140. F5 connects to HF TCP5140 but since TCP connections have both source and destination ports, F5 randomly needs to assign a source port for this connection (59697), which you see in the log files. It does not mean it is trying to connect to this 59697 port. This log is only for debugging inside firewall logs etc.

Listening directly TCP ports on Splunk is not recommended. Most probably your HF is having performance problems or cannot process data as fast as needed. At that time F5 detects that blocking and tries to restart the connection.

The best practice is using a Syslog server  (Rsylog or SyslogNG) to listen to TCP/UDP ports and writes to a file, HF monitors this file and ingests data. 

You can see below links for further information

https://conf.splunk.com/files/2017/slides/the-critical-syslog-tricks-that-no-one-seems-to-know-about...

Or you can use Splunk Connect for Syslog

https://splunk.github.io/splunk-connect-for-syslog/main/faq/

[GaetanVP]

Thanks for all those info !

One last question, when you mentioned " Syslog server  (Rsylog or SyslogNG) to listen to TCP/UDP ports and writes to a file, HF monitors this file and ingests data" 

Does the HF and the Syslog server would be on the same machine ?  

[scelikok]

Hi @GaetanVP,

TcpInputProc logs show the connection source in <src_ip_address>:<src_port> format. 59697 is the source port of your F5 device. Since it is retrying to connect HF to send data, this source port changes every time.

[GaetanVP]

Hello @scelikok, thanks but I am not sure to understand

How does the src port impact the data flow ?
I configured f5 to forward data to a specific port on  the HF, and I configured the HF to listen on that port.

Why would F5 retry to connect to HF with a different source port ? 

Thanks