Why do I get missing forwarders?

[las]

I have configured the Universal forwarder to monitor a folder, but when I look at the Splunkd.log on the forwarder I see the following:
09-07-2012 09:47:57.809 +0200 WARN TcpOutputProc - Cooked connection to ip=99.999.9.999:9997 timed out
09-07-2012 09:48:08.822 +0200 WARN TcpOutputFd - Connect to 99.999.9.999:9997 failed. No connection could be made because the target machine actively refused it.
09-07-2012 09:48:08.822 +0200 ERROR TcpOutputFd - Connection to host=99.999.9.999:9997 failed
09-07-2012 09:48:08.822 +0200 WARN TcpOutputProc - Applying quarantine to idx=99.999.9.999:9997 numberOfFailures=2

There are no log entries in the Splunkd.log on the indexer.
The universal forwarder is listed as missing in the Deployment App

The indexer is configured to listen for all servers on 9997, and I have seen data received on this port at an earlier time.

There are about 450 universel forwarders configured.

Kind regards.

[las · Accepted solution]

It wasn't a DoS attack, and windows didn't tag it that way.
There is a stanza connection_host = none in inputs.conf that solved the problem.

[MuS]

okay, good to know you solved your problem by setting the attribute connection_host = none

[MuS]

Hi las

my first guess was: 'about 450 universal forwarders configured'

... imagine they connect all at the same time to the indexer, your indexer could see this as a DoS attack.

Check your indexer and Server OS if there is some DoS detection happening.

/update:

I'm by far no TCP expert, but when you get 'a lot' of CLOSE_WAIT & FIN_WAIT_2 there is something not as it should. short example:

.1 The client sends a SYN to the server.
.2 The server responds with a SYN and ACK to the client.
.3 The client responds with an ACK to the server.

Connection is established and data is transfered (the steps are known as a 3 way handshake).
When the server is closing the connection, the following sequence takes place:

.4 The server sends a FIN and an ACK to the client.
.5 The client sends an ACK to the server.
.6 The client sends its own FIN and ACK to the server
.7 The server sends and ACK to the client.

The short version is that this state exists when the first FIN_ACK and ACK have been sent (steps 4 & 5) but the second FIN_ACK and ACK (steps 6 & 7) has not.
On the side that closed the connection you will have FIN_WAIT_2, on the side that is to send the final FIN_ACK and ACK you will have CLOSE_WAIT.

this could also be caused by a firewall somewhere in the network.
cheers,

MuS

[MuS]

see update....

[las]

I do have a lot of CLOSE_WAIT on the indexer, when I do a netstat, I have have seen FIN_WAIT_2 on the universal forwarder.
Does that support your theory?