How to fix Heavy Forwarder to Splunk Cloud logs forward error?

[mthirumalareddy]

I am using Heavy Forwarder to send logs from different sources such as Domain Controller, Windows Servers, Network Switches etc. to Splunk Cloud

I am receiving the following error from the Splunk Cloud

PeriodicHealthReporter - feature="TCPOutAutoLB-0" color=red indicator="s2s_connections" due_to_threshold_value=70 measured_value=100 reason="More than 70% of forwarding destinations have failed.  Ensure your hosts and ports in outputs.conf are correct.  Also ensure that the indexers are all running, and that any SSL certificates being used for forwarding are correct." node_type=indicator node_path=splunkd.data_forwarding.splunk-2-splunk_forwarding.tcpoutautolb-0.s2s_connections

Does anyone know what is the cause and how to fix it? 

[gcusello]

Hi @mthirumalareddy,

probably there some queue problem for network congestions or resources availability or wrong Splunk configurations.

You can identify the servers using a simple search like the following:

index=_internal  source=*metrics.log sourcetype=splunkd group=queue 
| eval name=case(name=="aggqueue","2 - Aggregation Queue",
 name=="indexqueue", "4 - Indexing Queue",
 name=="parsingqueue", "1 - Parsing Queue",
 name=="typingqueue", "3 - Typing Queue",
 name=="splunktcpin", "0 - TCP In Queue",
 name=="tcpin_cooked_pqueue", "0 - TCP In Queue") 
| eval max=if(isnotnull(max_size_kb),max_size_kb,max_size) 
| eval curr=if(isnotnull(current_size_kb),current_size_kb,current_size) 
| eval fill_perc=round((curr/max)*100,2) 
| bin _time span=1m
| stats Median(fill_perc) AS "fill_percentage" max(max) AS max max(curr) AS curr by host, _time, name 
| where (fill_percentage>70 AND name!="4 - Indexing Queue") OR (fill_percentage>70 AND name="4 - Indexing Queue")
| sort -_time

then you have to debug the problem,

I can help you only for for Splunk wrong configurations, you can setup the maxKBps=0 in limits.conf of the hosts with queues.

Then you should check if all your Heavy Forwarders, meet the hardware reference.

For network congestions, I cannot help you.

Ciao.

Giuseppe

[mthirumalareddy]

Hi @gcusello ,

We tried using the search that you provided and we see a little or zero fill percentage. I have attached the file for reference too. 

[gcusello]

Hi @mthirumalareddy,

check if there are high values for queues otherwise this means that this isn't the issue.

Ciao.

Giuseppe