How to fix TCPOutAutoLB-0 error?

[hketer]

Hello,

I'll try to explain our issue we had.

We have 7 HFs and 4 Idx
HF_1, HF_2, HF_3 sending TCP logs and log files to: HF_4 & HF_6 & HF_7 
HF_4 sending TCP logs (Not necessarily the same data) to HF_5 
HF_5 send the data from HF_4 to our Indexers.

The splunkd service on HF_5 was down, what cause our HF_4 to receive errors:

"TCPOutAutoLB-0 , forwarding destinations have failed"
make sense.

What I don't understand is why the servers HF_1\2\3 got stuck and stopped send data  also to HF_6 and HF_7.

Please help me understand this,
Thank you all!

Hen

[scelikok · Accepted solution]

Hi @hketer,

Are you also using TCP Syslog output on your HF_1/2/3 to any destination? Because if any of the Syslog TCP destinations gets stuck, all indexing and forwarding pipeline stops. That's why I prefer UDP Syslog output to prevent this problem. 

[hketer]

Hi @scelikok ,

Yes, we do use TCP Syslog outputs to other destinations.
As I understand, the queues will get stuck and will affect all the other inputs configured ?

Will it affect only TCP inputs? or also monitoring files for example?

Thanks,
Hen

[scelikok]

Hi @hketer,

Yes, unfortunately, it affects all indexing processes.

[hketer]

@scelikok 

Thank you for the help!