Yesterday we realized that three of our six production indexers stop listening on port 9997. We bounced them and all went back to normal. This morning, one of them stop listening on port 9997. Just bounced it and it's fine for now.
What can it be?
After bouncing -
$ netstat -plnt | grep 9997
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:9997 0.0.0.0:* LISTEN 10108/splunkd
index=_internal host=x1209 group=queue blocked name=indexqueue | timechart count by queue
I don't see the results matching the -
index=_internal host=x1209 group=queue unblocked name=indexqueue | timechart count by queue
Meaning, blocked versus unblocked.
My problem is that only by bouncing Splunk, the 9997 port becomes open and it starts indexing. According to the License Usage - Previous 30 Days, this indexer hasn't indexed any data for three days until the bounce.