Greetings,
A network engineer recently modified the firewalls for the ports and we are getting this error messages in our search head, Cluster Manager, and Indexers:
Auto Load Balanced TCP Output
Root Cause(s):
More than 70% of forwarding destinations have failed. Ensure your hosts and ports in outputs.conf are correct. Also ensure that the indexers are all running, and that any SSL certificates being used for forwarding are correct.
Well, the message is relatively self-explanatory. Well-configured Splunk environment should send their logs to the indexers instead of storing them locally. For some reason in your case this fails.
Check your outputs.conf settings,see where the events should be sent and verify why it doesn't work (network connectivity, failed components, expired ssl certs...)
Check the splunkd.log on each of those components for symptoms nd additional info.