topic Re: NOT a question: A heavy forwarder can be listening on port 9997 and still look like that port is down or blocked. in Getting Data In

NOT a question: A heavy forwarder can be listening on port 9997 and still look like that port is down or blocked.

lycollicott — Wed, 22 Feb 2017 20:58:40 GMT

First, some quick background about this tip.

Our Ops guys reported no recent events for their searches.
Universal Forwarders, Heavy Forwarders and Indexers were all up.
Those Ops guys were right! No recent events anywhere - not even _internal!
We cracked our knuckles and told them not to panic.

All these machines run Windows, so from a UF node we used Powershell to test the port on the HF:

$(new-object net.sockets.tcpclient).connect("10.xx.xx.xx",9997)

If that command is successful it will immediately return a good old C: prompt, but will throw an error afer a few seconds if it is unsuccessful. In our case it was unsuccessful. Grrr.

netstat -an showed that 9997 was listening on HF. Grrr.
Firewall guys said everything was cruising through unfettered. Grrr.

After growling for a bit and questioning the sanity of the firewall guys I looked at the indexer. Yup, it was running. Looked again and found this:

There was 9997 listening on the indexer...

PS C:\Windows\system32> netstat -an | findstr "9997"
  TCP    0.0.0.0:9997           0.0.0.0:0              LISTENING
  TCP    10.54.54.70:9997       10.54.52.85:60353      ESTABLISHED
  TCP    10.54.54.70:9997       10.54.54.32:52020      ESTABLISHED
  TCP    10.54.54.70:9997       10.54.54.32:52315      CLOSE_WAIT
  TCP    10.54.54.70:9997       10.54.54.33:51987      ESTABLISHED
  TCP    10.54.54.70:9997       10.54.54.33:52202      CLOSE_WAIT
  TCP    10.54.54.70:9997       10.54.54.33:52203      CLOSE_WAIT
  TCP    10.54.54.70:9997       10.54.54.34:63000      ESTABLISHED

But wait a minute....it isn't.....

PS C:\Windows\system32> netstat -an | findstr "LISTEN"
      TCP    0.0.0.0:135            0.0.0.0:0              LISTENING
      TCP    0.0.0.0:445            0.0.0.0:0              LISTENING
      TCP    0.0.0.0:3389           0.0.0.0:0              LISTENING
      TCP    0.0.0.0:5985           0.0.0.0:0              LISTENING
      TCP    0.0.0.0:8089           0.0.0.0:0              LISTENING
      TCP    0.0.0.0:8191           0.0.0.0:0              LISTENING
      TCP    0.0.0.0:9887           0.0.0.0:0              LISTENING
      TCP    0.0.0.0:10000          0.0.0.0:0              LISTENING
      TCP    0.0.0.0:47001          0.0.0.0:0              LISTENING
      TCP    0.0.0.0:49152          0.0.0.0:0              LISTENING
      TCP    0.0.0.0:49153          0.0.0.0:0              LISTENING
      TCP    0.0.0.0:49154          0.0.0.0:0              LISTENING
      TCP    0.0.0.0:49155          0.0.0.0:0              LISTENING
      TCP    0.0.0.0:49183          0.0.0.0:0              LISTENING
      TCP    0.0.0.0:49198          0.0.0.0:0              LISTENING

.
Well.

So, the heavy forwarder accepted my incoming Powershell connection and routed that connection right over to the indexer where it failed. I bounced the indexer and like magic it was fixed.

I like to share the strange, silly and stupid things I notice, so maybe this will help someone somewhere keep from staring at their screen in confusion for 30 minutes like I did today.

Re: NOT a question: A heavy forwarder can be listening on port 9997 and still look like that port is down or blocked.

lycollicott — Wed, 22 Feb 2017 20:59:34 GMT

Just sharing. 🙂

Re: NOT a question: A heavy forwarder can be listening on port 9997 and still look like that port is down or blocked.

woodcock — Thu, 23 Feb 2017 07:26:14 GMT

Let me guess: The Indexers are Windows OS, right?

Re: NOT a question: A heavy forwarder can be listening on port 9997 and still look like that port is down or blocked.

lycollicott — Thu, 23 Feb 2017 13:27:32 GMT

Oh, yeah. Wheeeeeee.

I think I am slowly making headway on a campaign to go Linux. Wish me luck!

Re: NOT a question: A heavy forwarder can be listening on port 9997 and still look like that port is down or blocked.

ddrillic — Thu, 23 Feb 2017 14:40:50 GMT

We had recently the same behavior on Linux for the indexers. The situation was so severe that we ended up creating a monitoring page for port 9997 for all the indexers. If the port is blocked and doesn't get open in a timely manner, our procedure is to bounce the indexer.

The root cause, in our case, was the fact that the indexing queues were filled up and by making them bigger, the situation is much better.

We need probably to open an enhancement request for this type of behavior.

Re: NOT a question: A heavy forwarder can be listening on port 9997 and still look like that port is down or blocked.

woodcock — Thu, 23 Feb 2017 14:52:41 GMT

And you skipped your monthly reboot!

Re: NOT a question: A heavy forwarder can be listening on port 9997 and still look like that port is down or blocked.

lycollicott — Thu, 23 Feb 2017 16:46:59 GMT

LOL it was rebooted exactly 14 days earlier.