Getting Data In

How to fix TCPOutAutoLB-0 error?

hketer
Path Finder

Hello,

I'll try to explain our issue we had.

We have 7 HFs and 4 Idx
HF_1, HF_2, HF_3 sending TCP logs and log files to: HF_4 & HF_6 & HF_7 
HF_4 sending TCP logs (Not necessarily the same data) to HF_5 
HF_5 send the data from HF_4 to our Indexers.

 

The splunkd service on HF_5 was down, what cause our HF_4 to receive errors:

"TCPOutAutoLB-0 , forwarding destinations have failed"
make sense.

What I don't understand is why the servers HF_1\2\3 got stuck and stopped send data  also to HF_6 and HF_7.

 

Please help me understand this,
Thank you all!

 

Hen

 

 

 

Labels (1)
0 Karma
1 Solution

scelikok
SplunkTrust
SplunkTrust

Hi @hketer,

Are you also using TCP Syslog output on your HF_1/2/3 to any destination? Because if any of the Syslog TCP destinations gets stuck, all indexing and forwarding pipeline stops. That's why I prefer UDP Syslog output to prevent this problem. 

If this reply helps you an upvote and "Accept as Solution" is appreciated.

View solution in original post

scelikok
SplunkTrust
SplunkTrust

Hi @hketer,

Are you also using TCP Syslog output on your HF_1/2/3 to any destination? Because if any of the Syslog TCP destinations gets stuck, all indexing and forwarding pipeline stops. That's why I prefer UDP Syslog output to prevent this problem. 

If this reply helps you an upvote and "Accept as Solution" is appreciated.

hketer
Path Finder

Hi @scelikok ,

Yes, we do use TCP Syslog outputs to other destinations.
As I understand, the queues will get stuck and will affect all the other inputs configured ?

Will it affect only TCP inputs? or also monitoring files for example?

Thanks,
Hen

0 Karma

scelikok
SplunkTrust
SplunkTrust

Hi @hketer,

Yes, unfortunately, it affects all indexing processes.

If this reply helps you an upvote and "Accept as Solution" is appreciated.

hketer
Path Finder

@scelikok 

Thank you for the help!

Get Updates on the Splunk Community!

Technical Workshop Series: Splunk Data Management and SPL2 | Register here!

Hey, Splunk Community! Ready to take your data management skills to the next level? Join us for a 3-part ...

Spotting Financial Fraud in the Haystack: A Guide to Behavioral Analytics with Splunk

In today's digital financial ecosystem, security teams face an unprecedented challenge. The sheer volume of ...

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability As businesses scale ...