Getting Data In

Why some of our indexers stop listening on port 9997?

ddrillic
Ultra Champion

Yesterday we realized that three of our six production indexers stop listening on port 9997. We bounced them and all went back to normal. This morning, one of them stop listening on port 9997. Just bounced it and it's fine for now.

What can it be?

After bouncing -

$ netstat  -plnt | grep 9997
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:9997            0.0.0.0:*               LISTEN      10108/splunkd
Tags (2)
1 Solution

phoffman_splunk
Splunk Employee
Splunk Employee

when your queues are blocked and the blocking reaches back to input queue the indexer will stop listening to give it time to catch up.

http://docs.splunk.com/Documentation/Splunk/6.5.1/Forwarding/Receiverconnection#Closed_receiver_sock...

View solution in original post

phoffman_splunk
Splunk Employee
Splunk Employee

when your queues are blocked and the blocking reaches back to input queue the indexer will stop listening to give it time to catch up.

http://docs.splunk.com/Documentation/Splunk/6.5.1/Forwarding/Receiverconnection#Closed_receiver_sock...

View solution in original post

ddrillic
Ultra Champion

Closed receiver socket

When I run the following -

index=_internal host=x1209 group=queue blocked name=indexqueue | timechart count by queue

I don't see the results matching the -

index=_internal host=x1209 group=queue unblocked name=indexqueue | timechart count by queue

Meaning, blocked versus unblocked.

My problem is that only by bouncing Splunk, the 9997 port becomes open and it starts indexing. According to the License Usage - Previous 30 Days, this indexer hasn't indexed any data for three days until the bounce.

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!