Getting Data In

FwdDataReceiverThread error | Read Timeout Timed out after 600 seconds

GaetanVP
Communicator

Hello Splunkers,

I the following error on my Splunk HF which is listening to incoming data from F5 network appliance.

 

01-25-2023 08:06:56.794 +0000 ERROR TcpInputProc [2612981 FwdDataReceiverThread] - Error encountered for connection from src=<internal_ip_f5>:59697. Read Timeout Timed out after 600 seconds.

 

I am wondering what the number after the F5 IP is... I specified a unique port for the forwarding of data between f5 and the HF so I do not understand why I have number like 59697 (and many others).

More generally I do not know how to troubleshoot this...

Thanks for your help,

GaetanVP

Labels (2)
0 Karma
1 Solution

scelikok
SplunkTrust
SplunkTrust

Hi @GaetanVP,

Let's assume you configured HF to listen TCP 5140 port and configured F5 to send to TCP 5140. F5 connects to HF TCP5140 but since TCP connections have both source and destination ports, F5 randomly needs to assign a source port for this connection (59697), which you see in the log files. It does not mean it is trying to connect to this 59697 port. This log is only for debugging inside firewall logs etc.

Listening directly TCP ports on Splunk is not recommended. Most probably your HF is having performance problems or cannot process data as fast as needed. At that time F5 detects that blocking and tries to restart the connection.

The best practice is using a Syslog server  (Rsylog or SyslogNG) to listen to TCP/UDP ports and writes to a file, HF monitors this file and ingests data. 

You can see below links for further information

https://conf.splunk.com/files/2017/slides/the-critical-syslog-tricks-that-no-one-seems-to-know-about...

Or you can use Splunk Connect for Syslog

https://splunk.github.io/splunk-connect-for-syslog/main/faq/

 

If this reply helps you an upvote is appreciated.

View solution in original post

scelikok
SplunkTrust
SplunkTrust

Hi @GaetanVP,

Yes, they work on the same machine. Syslog server writes logs to the filesystem, on the same machine HF or UF will use monitor input to ingest the data.

 

If this reply helps you an upvote is appreciated.

scelikok
SplunkTrust
SplunkTrust

Hi @GaetanVP,

Let's assume you configured HF to listen TCP 5140 port and configured F5 to send to TCP 5140. F5 connects to HF TCP5140 but since TCP connections have both source and destination ports, F5 randomly needs to assign a source port for this connection (59697), which you see in the log files. It does not mean it is trying to connect to this 59697 port. This log is only for debugging inside firewall logs etc.

Listening directly TCP ports on Splunk is not recommended. Most probably your HF is having performance problems or cannot process data as fast as needed. At that time F5 detects that blocking and tries to restart the connection.

The best practice is using a Syslog server  (Rsylog or SyslogNG) to listen to TCP/UDP ports and writes to a file, HF monitors this file and ingests data. 

You can see below links for further information

https://conf.splunk.com/files/2017/slides/the-critical-syslog-tricks-that-no-one-seems-to-know-about...

Or you can use Splunk Connect for Syslog

https://splunk.github.io/splunk-connect-for-syslog/main/faq/

 

If this reply helps you an upvote is appreciated.

GaetanVP
Communicator

Thanks for all those info !

One last question, when you mentioned " Syslog server  (Rsylog or SyslogNG) to listen to TCP/UDP ports and writes to a file, HF monitors this file and ingests data" 

Does the HF and the Syslog server would be on the same machine ?  

0 Karma

scelikok
SplunkTrust
SplunkTrust

Hi @GaetanVP,

TcpInputProc logs show the connection source in <src_ip_address>:<src_port> format. 59697 is the source port of your F5 device. Since it is retrying to connect HF to send data, this source port changes every time.

If this reply helps you an upvote is appreciated.

GaetanVP
Communicator

Hello @scelikok, thanks but I am not sure to understand

How does the src port impact the data flow ?
I configured f5 to forward data to a specific port on  the HF, and I configured the HF to listen on that port.

Why would F5 retry to connect to HF with a different source port ? 

Thanks

0 Karma
Get Updates on the Splunk Community!

Dashboard Studio Challenge - Learn New Tricks, Showcase Your Skills, and Win Prizes!

Reimagine what you can do with your dashboards. Dashboard Studio is Splunk’s newest dashboard builder to ...

Introducing Edge Processor: Next Gen Data Transformation

We get it - not only can it take a lot of time, money and resources to get data into Splunk, but it also takes ...

Take the 2021 Splunk Career Survey for $50 in Amazon Cash

Help us learn about how Splunk has impacted your career by taking the 2021 Splunk Career Survey. Last year’s ...