How can I fix the issue related to More than 70% of forwarding destinations have failed

[MrBLeu]

01-09-2025 17:30:30.169 -0500 INFO  PeriodicHealthReporter - feature="TCPOutAutoLB-0" color=red indicator="s2s_connections" due_to_threshold_value=70 measured_value=100 reason="More than 70% of forwarding destinations have failed.  Ensure your hosts and ports in outputs.conf are correct.  Also ensure that the indexers are all running, and that any SSL certificates being used for forwarding are correct." node_type=indicator node_path=splunkd.data_forwarding.splunk-2-splunk_forwarding.tcpoutautolb-0.s2s_connections

[kiran_panchavat]

@MrBLeu 

If SSL is being used, ...

To do an openssl test like

openssl s_client -connect xx.xx.xx.xx:9997 -cert <cert_file> -CAfile <ca_file>

You can get <ca_file> from running this:

/opt/splunk/bin/splunk cmd btool server list sslConfig | grep sslRootCAPath

<cert_file> you can get from running this:

/opt/splunk/bin/splunk cmd btool outputs list tcpout

You are looking for the clientCert setting. If you have multiple entries for clientCert, such as one under [tcpout] and one under [tcpout:<group>], pick the one on the latter, which would be at the more specific level.
You'll be able to see if ssl handshake is completing properly with the settings currently configured.

[gcusello]

Hi @MrBLeu ,

at first, check if your UFs send data or not and check what are the Indexers receivers.

Then check all the connections from the UFs to the Indexers, maybe there are some closed connections.

Then are you using an SSL certificate?

if yes, check the validiti and the password of your certificate and that the certificate is used bonth on UFs and IDXs.

Ciao,

Giuseppe

[kiran_panchavat]

@MrBLeu  

Did you check your resource usage ?
What about network connections ?
Check the _internal logs on your server

I hope this helps, if any reply helps you, you could add your upvote/karma points to that reply, thanks.