topic Re: Why is heavy forwarder repeatedly getting "WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 600 seconds." in Getting Data In

Why is heavy forwarder repeatedly getting "WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 600 seconds."

hagjos43 — Tue, 09 Sep 2014 11:57:19 GMT

We are seeing the following errors on our Heavy Forwarder side:

09-05-2014 13:39:06.483 - 0400 INFO TcpOutputProc - Connected to idx= 23.42.214.219:9997
09-05-2014 13:39:06.484 - 0400 WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 600 seconds.
09-05-2014 13:39:36.493 - 0400 INFO TcpOutputProc - Connected to idx= 23.42.214.219:9997
09-05-2014 13:40:06.501 - 0400 INFO TcpOutputProc - Connected to idx= 23.42.214.219:9997
09-05-2014 13:40:36.509 - 0400 INFO TcpOutputProc - Connected to idx= 23.42.214.219:9997
09-05-2014 13:40:39.510 - 0400 WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 700 seconds.
09-05-2014 13:41:06.517 - 0400 INFO TcpOutputProc - Connected to idx= 23.42.214.219:9997
09-05-2014 13:41:36.524 - 0400 INFO TcpOutputProc - Connected to idx= 23.42.214.219:9997
09-05-2014 13:42:06.533 - 0400 INFO TcpOutputProc - Connected to idx= 23.42.214.219:9997
09-05-2014 13:42:19.536 - 0400 WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 800 seconds.

This continues to repeat through the current date. Anyone else experience this or have any suggestions?

Re: Why is heavy forwarder repeatedly getting "WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 600 seconds."

cdupuis123 — Mon, 12 Jan 2015 17:34:25 GMT

I don't have the answer, but I've got the same issue!!!! Anyone????

Re: Why is heavy forwarder repeatedly getting "WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 600 seconds."

djfisher — Fri, 20 Feb 2015 21:10:30 GMT

Same here,, started happening. Is it due to bad band width or to many seconds between collections? I use the 9*Nix app to collect audit logs using rlog.sh

Re: Why is heavy forwarder repeatedly getting "WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 600 seconds."

inters — Tue, 03 Mar 2015 17:37:30 GMT

I am currently evaluating Splunk. Ceaselessly, I encounter errors like this and "answers.splunk.com" has no answers, only other frustrated questioners.

Why does anyone use this software???

Re: Why is heavy forwarder repeatedly getting "WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 600 seconds."

satishsdange — Wed, 04 Mar 2015 07:37:56 GMT

Please post your questions..I am sure you will get answers.

Re: Why is heavy forwarder repeatedly getting "WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 600 seconds."

cdupuis123 — Thu, 05 Mar 2015 10:36:43 GMT

Hi inters

Yes I've spent time on the answers site with similar results, but after using/running Splunk now for 3 years I've found that if I can't get the answer from Splunk answers I've either used the wrong search term, or most times I find something close and am able to backwards/sideways engineer it until it fixes my issue. Oh course if all else fails call my SE or Support. Good luck with your POC

Re: Why is heavy forwarder repeatedly getting "WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 600 seconds."

khourihan_splun — Mon, 28 Sep 2015 16:48:13 GMT

See this post for step to troubleshoot: http://answers.splunk.com/answers/189238/how-to-troubleshoot-error-on-splunk-6-universal-fo.html

but in general I'd use Splunk on Splunk (SoS) app to diagnose where the bottleneck is. If you are running 6.3, you can use the DMC (Distributed Management Console) to do the same analysis: Goto Setting and click Distribute Management Console icon on the left.

Re: Why is heavy forwarder repeatedly getting "WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 600 seconds."

bohanlon_splunk — Fri, 29 Jan 2016 16:36:45 GMT

Have you a Forwarder Loop?
https://answers.splunk.com/answers/217915/splunk-app-for-windows-infrastructure-forwarding-t.html

Re: Why is heavy forwarder repeatedly getting "WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 600 seconds."

masonmorales — Fri, 29 Jan 2016 18:45:54 GMT

From my experience, this is usually due to blocked queues at the indexers. The most common cause is insufficient IOPS/throughput at the indexers' disk subsystem. When a queue is full for a certain length of time on the indexer, the indexer will start rejecting forwarder connections so that it can clear its full queue(s) before processing new events.

Here are some searches you can run against the _internal index of your indexers to find and see the bottleneck:

View the current queue size:

index=_internal source=*metrics.log group=queue | timechart median(current_size) by name

Find blocked queue events:

index=_internal source=*metrics.log group=queue blocked
Blocked queues in last 24 hours by queue and Splunk server: 
index=_internal source=*metrics.log sourcetype=splunkd group=queue | eval max=if(isnotnull(max_size_kb),max_size_kb,max_size)  | eval curr=if(isnotnull(current_size_kb),current_size_kb,current_size)  | eval fill_perc=round((curr/max)*100,2) | eval name=host.":".name | where fill_perc>=99.0 | timechart max(fill_perc) as MaxFillPerc by name useother=false limit=100 minspan=1h

Count how many times queues were >=99% by Queue Name and Splunk Server

index=_internal source=*metrics.log sourcetype=splunkd group=queue | eval max=if(isnotnull(max_size_kb),max_size_kb,max_size) | eval curr=if(isnotnull(current_size_kb),current_size_kb,current_size)  | eval fill_perc=round((curr/max)*100,2) | where fill_perc>=99.0 | stats count by name host  | eval name=case(name=="aggqueue","2 - Aggregation Queue",name=="indexqueue","4 - Indexing Queue",name=="parsingqueue","1 - Parsing Queue",name=="typingqueue","3 - Typing Queue", 1=1, name)

Re: Why is heavy forwarder repeatedly getting "WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 600 seconds."

PGrantham — Fri, 29 Jan 2016 18:49:14 GMT

Try checking your metrics.log on both your HF and indexer.

Do you see any blocked queues (like the parsingqueue or aggqueue or tcpinqueue)?