topic Why some of our indexers stop listening on port 9997? in Getting Data In

Why some of our indexers stop listening on port 9997?

ddrillic — Tue, 24 Jan 2017 14:20:26 GMT

Yesterday we realized that three of our six production indexers stop listening on port 9997. We bounced them and all went back to normal. This morning, one of them stop listening on port 9997. Just bounced it and it's fine for now.

What can it be?

After bouncing -

$ netstat  -plnt | grep 9997
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:9997            0.0.0.0:*               LISTEN      10108/splunkd

Re: Why some of our indexers stop listening on port 9997?

phoffman_splunk — Tue, 29 Sep 2020 12:35:21 GMT

when your queues are blocked and the blocking reaches back to input queue the indexer will stop listening to give it time to catch up.

http://docs.splunk.com/Documentation/Splunk/6.5.1/Forwarding/Receiverconnection#Closed_receiver_socket

Re: Why some of our indexers stop listening on port 9997?

ddrillic — Tue, 24 Jan 2017 19:24:09 GMT

When I run the following -

index=_internal host=x1209 group=queue blocked name=indexqueue | timechart count by queue

I don't see the results matching the -

index=_internal host=x1209 group=queue unblocked name=indexqueue | timechart count by queue

Meaning, blocked versus unblocked.

My problem is that only by bouncing Splunk, the 9997 port becomes open and it starts indexing. According to the License Usage - Previous 30 Days, this indexer hasn't indexed any data for three days until the bounce.