Search Head Error : Forwarding to indexer group primary_indexers blocked

[somesoni2]

Hello Splunkers,

I have 4 Indexers in a cluster, serving 2 search heads, both stand-alone (no clustering/pooling), with one acting as job server (this is where I run my summary index searches).

From Past two days, I’m seeing this banner message on the Job server search head,

*Search peer –SHHostNameHere-- has the following message: Forwarding to indexer group primary_indexers blocked for 1900 seconds.*

The summary index data is not flowing to the Indexers and is stored locally on the $Splunk_Home/var/run/spool/splunk folder. Please note that other forwarders are sending data without any issues.

I also see following entries from _internal logs for source=metrics.log

08-05-2015 15:14:19.449 -0400 INFO Metrics - group=queue, name=indexqueue, blocked=true, max_size_kb=10240, current_size_kb=10239, current_size=25976, largest_size=25976, smallest_size=608

I got these messages for indexqueue (98%), and very little count for typingqueue, aggqueue auditqueue and parsingqueue.

After reading some posts here, I have tried following so far.

1. Indexers – added following and restarted

   [splunktcp://9997]
   connection_host = ip

2. Job Server SearchHead – added following to server.conf and restared.

   [queue]
   maxSize = 20MB

   [queue=indexQueue]
   maxSize = 10MB

   [queue=parsingQueue]
   maxSize = 10MB

With all these options as well, I can still see that error message and issue.

Does anyone here has faced similar issues and have some more options for me to try out. The summary index data feeds lots of dashboards so need to get this resolved soon.

[somesoni2 · Accepted solution]

Along with the server.conf settings, I have done the TCP Tuning on the Job Server, (/etc/sysctl.conf file changes, reference http://kaivanov.blogspot.com/2010/09/linux-tcp-tuning.html), and that seems to have resolve the blocking. I will continue to test for some time to if this comes back again.

[somesoni2]

It did work fine.

[martin_mueller]

Is indexers:9997 reachable from the job server?
Does the job server appear in this search result on the indexers?

index=_internal group=tcpin_connections

[somesoni2]

@Martin,

The job server do appear in the search result of the query (in the field sourceHost).
The job server was working fine 2 days back. The only thing that has changed is the amount of data for summary indexing has been increased (as the production rollout is happening).

[lkoppolu]

@somesoni2

I got the same problem in my Local Laptop running on Windows OS (I use this setup for experimenting Splunk features), I tried configuring same way, somewhere it went wrong configuration and got the same error message.

[vasanthmss]

Search helps to find the max queue size which is blocked in indexer or forwarder,

index=_internal  sourcetype=splunkd blocked=true | stats values(max_size_kb) as kb by name,host | sort 0 name | eval mb=kb/1024

index=_internal source=*metrics.log group=queue "blocked" | transaction blocked name host | eval queue_blocked = duration /60 | where queue_blocked > 1 | table host name blocked queue_blocked

Solutions

Queue size increase in forwarder does impact on the indexers (forwarder queue change will helps till parsing queue), So I guess we can apply the same conf in indexers and test.
(Note: Reason for queue block is, when some component in the index time can not service data as fast is entering into the system)

Sources : http://wiki.splunk.com/Community:HowIndexingWorks