Auto Load Balanced TCP Output (error)

[KayBeesKnees83]

Greetings,

A network engineer recently modified the firewalls for the ports and we are getting this error messages in our search head, Cluster Manager, and Indexers: 

Auto Load Balanced TCP Output

Root Cause(s):

More than 70% of forwarding destinations have failed. Ensure your hosts and ports in outputs.conf are correct. Also ensure that the indexers are all running, and that any SSL certificates being used for forwarding are correct.

[PickleRick]

Well, the message is relatively self-explanatory. Well-configured Splunk environment should send their logs to the indexers instead of storing them locally. For some reason in your case this fails.

Check your outputs.conf settings,see where the events should be sent and verify why it doesn't work (network connectivity, failed components, expired ssl certs...)

Check the splunkd.log on each of those components for symptoms nd additional info.

[nqjpm]

I had this same problem so I thought I would share. For me, I was dealing with a clustered Environment. I went and looked at Splunkd.log and saw a bunch of messages like network unreachable and could not connect to peer etc. 

Turned out Splunkd was down.  Restarted Splunkd on the CM and it reconnected. Hope that helps someone in the future.