topic Re: Why do I get missing forwarders? in Getting Data In

Why do I get missing forwarders?

las — Fri, 07 Sep 2012 08:18:07 GMT

I have configured the Universal forwarder to monitor a folder, but when I look at the Splunkd.log on the forwarder I see the following:
09-07-2012 09:47:57.809 +0200 WARN TcpOutputProc - Cooked connection to ip=99.999.9.999:9997 timed out
09-07-2012 09:48:08.822 +0200 WARN TcpOutputFd - Connect to 99.999.9.999:9997 failed. No connection could be made because the target machine actively refused it.
09-07-2012 09:48:08.822 +0200 ERROR TcpOutputFd - Connection to host=99.999.9.999:9997 failed
09-07-2012 09:48:08.822 +0200 WARN TcpOutputProc - Applying quarantine to idx=99.999.9.999:9997 numberOfFailures=2

There are no log entries in the Splunkd.log on the indexer.
The universal forwarder is listed as missing in the Deployment App

The indexer is configured to listen for all servers on 9997, and I have seen data received on this port at an earlier time.

There are about 450 universel forwarders configured.

Kind regards.

Re: Why do I get missing forwarders?

MuS — Fri, 07 Sep 2012 08:25:34 GMT

Hi las

my first guess was: 'about 450 universal forwarders configured'

... imagine they connect all at the same time to the indexer, your indexer could see this as a DoS attack.

Check your indexer and Server OS if there is some DoS detection happening.

/update:

I'm by far no TCP expert, but when you get 'a lot' of CLOSE_WAIT & FIN_WAIT_2 there is something not as it should. short example:

.1 The client sends a SYN to the server.
.2 The server responds with a SYN and ACK to the client.
.3 The client responds with an ACK to the server.

Connection is established and data is transfered (the steps are known as a 3 way handshake).
When the server is closing the connection, the following sequence takes place:

.4 The server sends a FIN and an ACK to the client.
.5 The client sends an ACK to the server.
.6 The client sends its own FIN and ACK to the server
.7 The server sends and ACK to the client.

The short version is that this state exists when the first FIN_ACK and ACK have been sent (steps 4 & 5) but the second FIN_ACK and ACK (steps 6 & 7) has not.
On the side that closed the connection you will have FIN_WAIT_2, on the side that is to send the final FIN_ACK and ACK you will have CLOSE_WAIT.

this could also be caused by a firewall somewhere in the network.
cheers,

MuS

Re: Why do I get missing forwarders?

las — Mon, 28 Sep 2020 12:23:52 GMT

I do have a lot of CLOSE_WAIT on the indexer, when I do a netstat, I have have seen FIN_WAIT_2 on the universal forwarder.
Does that support your theory?

Re: Why do I get missing forwarders?

MuS — Fri, 07 Sep 2012 09:00:47 GMT

see update....

Re: Why do I get missing forwarders?

las — Fri, 14 Sep 2012 08:20:39 GMT

It wasn't a DoS attack, and windows didn't tag it that way.
There is a stanza connection_host = none in inputs.conf that solved the problem.

Re: Why do I get missing forwarders?

MuS — Fri, 14 Sep 2012 08:35:18 GMT

okay, good to know you solved your problem by setting the attribute connection_host = none