Connection errors to heavy forwarders

[ebaileytu]

we have the following setup

2 heavy forwarders (HF) forwarding data to 4 indexers

We just added another 100 Universal forwarders (UF) to the environment so now we have about 800 UFs connecting to the HFs. I am starting to see a troubling number of connection error messages (about 7000 per hour) from the UFs such as:

05-20-2014 21:10:16.949 -0500 ERROR TcpOutputFd - Connection to host=xx.xxx.xx.xx:xxxx failed. sock_error = 10054. SSL Error = error:00000000:lib(0):func(0):reason(0)

(We are using SSL for connections from the UF to HF)

and

05-20-2014 21:09:59.394 -0500 ERROR TcpOutputFd - Connection to host=xx.xxx.xx.xx:xxxx failed

Data is getting forwarded from the UF to the HF but from tests I can see some data is delayed. Do the errors indicate I need to adjust a setting or just deploy another HF? I do not see high resource utilization on the HF.

Thanks!

[ebaileytu · Accepted solution]

issue was with the ESX server hosting the HF - very high iowait was the issue

[gsopko]

Hi, what was the solution? 🙂

Thanks

[ebaileytu]

issue with ESX server storage - high iowait created chaos