we have the following setup
2 heavy forwarders (HF) forwarding data to 4 indexers
We just added another 100 Universal forwarders (UF) to the environment so now we have about 800 UFs connecting to the HFs. I am starting to see a troubling number of connection error messages (about 7000 per hour) from the UFs such as:
05-20-2014 21:10:16.949 -0500 ERROR TcpOutputFd - Connection to host=xx.xxx.xx.xx:xxxx failed. sock_error = 10054. SSL Error = error:00000000:lib(0):func(0):reason(0)
(We are using SSL for connections from the UF to HF)
05-20-2014 21:09:59.394 -0500 ERROR TcpOutputFd - Connection to host=xx.xxx.xx.xx:xxxx failed
Data is getting forwarded from the UF to the HF but from tests I can see some data is delayed. Do the errors indicate I need to adjust a setting or just deploy another HF? I do not see high resource utilization on the HF.