Why some of our indexers stop listening on port 9997?

[ddrillic]

Yesterday we realized that three of our six production indexers stop listening on port 9997. We bounced them and all went back to normal. This morning, one of them stop listening on port 9997. Just bounced it and it's fine for now.

What can it be?

After bouncing -

$ netstat  -plnt | grep 9997
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:9997            0.0.0.0:*               LISTEN      10108/splunkd

[phoffman_splunk · Accepted solution]

when your queues are blocked and the blocking reaches back to input queue the indexer will stop listening to give it time to catch up.

http://docs.splunk.com/Documentation/Splunk/6.5.1/Forwarding/Receiverconnection#Closed_receiver_sock...

[ddrillic]

When I run the following -

index=_internal host=x1209 group=queue blocked name=indexqueue | timechart count by queue

I don't see the results matching the -

index=_internal host=x1209 group=queue unblocked name=indexqueue | timechart count by queue

Meaning, blocked versus unblocked.

My problem is that only by bouncing Splunk, the 9997 port becomes open and it starts indexing. According to the License Usage - Previous 30 Days, this indexer hasn't indexed any data for three days until the bounce.