Getting Data In

FwdDataReceiverThread error | Read Timeout Timed out after 600 seconds

GaetanVP
Communicator

Hello Splunkers,

I the following error on my Splunk HF which is listening to incoming data from F5 network appliance.

 

01-25-2023 08:06:56.794 +0000 ERROR TcpInputProc [2612981 FwdDataReceiverThread] - Error encountered for connection from src=<internal_ip_f5>:59697. Read Timeout Timed out after 600 seconds.

 

I am wondering what the number after the F5 IP is... I specified a unique port for the forwarding of data between f5 and the HF so I do not understand why I have number like 59697 (and many others).

More generally I do not know how to troubleshoot this...

Thanks for your help,

GaetanVP

Labels (2)
0 Karma
1 Solution

scelikok
SplunkTrust
SplunkTrust

Hi @GaetanVP,

Let's assume you configured HF to listen TCP 5140 port and configured F5 to send to TCP 5140. F5 connects to HF TCP5140 but since TCP connections have both source and destination ports, F5 randomly needs to assign a source port for this connection (59697), which you see in the log files. It does not mean it is trying to connect to this 59697 port. This log is only for debugging inside firewall logs etc.

Listening directly TCP ports on Splunk is not recommended. Most probably your HF is having performance problems or cannot process data as fast as needed. At that time F5 detects that blocking and tries to restart the connection.

The best practice is using a Syslog server  (Rsylog or SyslogNG) to listen to TCP/UDP ports and writes to a file, HF monitors this file and ingests data. 

You can see below links for further information

https://conf.splunk.com/files/2017/slides/the-critical-syslog-tricks-that-no-one-seems-to-know-about...

Or you can use Splunk Connect for Syslog

https://splunk.github.io/splunk-connect-for-syslog/main/faq/

 

If this reply helps you an upvote is appreciated.

View solution in original post

scelikok
SplunkTrust
SplunkTrust

Hi @GaetanVP,

Yes, they work on the same machine. Syslog server writes logs to the filesystem, on the same machine HF or UF will use monitor input to ingest the data.

 

If this reply helps you an upvote is appreciated.

scelikok
SplunkTrust
SplunkTrust

Hi @GaetanVP,

Let's assume you configured HF to listen TCP 5140 port and configured F5 to send to TCP 5140. F5 connects to HF TCP5140 but since TCP connections have both source and destination ports, F5 randomly needs to assign a source port for this connection (59697), which you see in the log files. It does not mean it is trying to connect to this 59697 port. This log is only for debugging inside firewall logs etc.

Listening directly TCP ports on Splunk is not recommended. Most probably your HF is having performance problems or cannot process data as fast as needed. At that time F5 detects that blocking and tries to restart the connection.

The best practice is using a Syslog server  (Rsylog or SyslogNG) to listen to TCP/UDP ports and writes to a file, HF monitors this file and ingests data. 

You can see below links for further information

https://conf.splunk.com/files/2017/slides/the-critical-syslog-tricks-that-no-one-seems-to-know-about...

Or you can use Splunk Connect for Syslog

https://splunk.github.io/splunk-connect-for-syslog/main/faq/

 

If this reply helps you an upvote is appreciated.

GaetanVP
Communicator

Thanks for all those info !

One last question, when you mentioned " Syslog server  (Rsylog or SyslogNG) to listen to TCP/UDP ports and writes to a file, HF monitors this file and ingests data" 

Does the HF and the Syslog server would be on the same machine ?  

0 Karma

scelikok
SplunkTrust
SplunkTrust

Hi @GaetanVP,

TcpInputProc logs show the connection source in <src_ip_address>:<src_port> format. 59697 is the source port of your F5 device. Since it is retrying to connect HF to send data, this source port changes every time.

If this reply helps you an upvote is appreciated.

GaetanVP
Communicator

Hello @scelikok, thanks but I am not sure to understand

How does the src port impact the data flow ?
I configured f5 to forward data to a specific port on  the HF, and I configured the HF to listen on that port.

Why would F5 retry to connect to HF with a different source port ? 

Thanks

0 Karma
Get Updates on the Splunk Community!

New Splunk Observability innovations: Deeper visibility and smarter alerting to ...

You asked, we delivered. Splunk Observability Cloud has several new innovations giving you deeper visibility ...

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...