Getting Data In

Connection errors to heavy forwarders

ebaileytu
Communicator

we have the following setup

2 heavy forwarders (HF) forwarding data to 4 indexers

We just added another 100 Universal forwarders (UF) to the environment so now we have about 800 UFs connecting to the HFs. I am starting to see a troubling number of connection error messages (about 7000 per hour) from the UFs such as:

05-20-2014 21:10:16.949 -0500 ERROR TcpOutputFd - Connection to host=xx.xxx.xx.xx:xxxx failed. sock_error = 10054. SSL Error = error:00000000:lib(0):func(0):reason(0)

(We are using SSL for connections from the UF to HF)

and

05-20-2014 21:09:59.394 -0500 ERROR TcpOutputFd - Connection to host=xx.xxx.xx.xx:xxxx failed

Data is getting forwarded from the UF to the HF but from tests I can see some data is delayed. Do the errors indicate I need to adjust a setting or just deploy another HF? I do not see high resource utilization on the HF.

Thanks!

Tags (2)
0 Karma
1 Solution

ebaileytu
Communicator

issue was with the ESX server hosting the HF - very high iowait was the issue

View solution in original post

ebaileytu
Communicator

issue was with the ESX server hosting the HF - very high iowait was the issue

gsopko
New Member

Hi, what was the solution? 🙂

Thanks

0 Karma

ebaileytu
Communicator

issue with ESX server storage - high iowait created chaos

0 Karma
Get Updates on the Splunk Community!

Prove Your Splunk Prowess at .conf25—No Prereqs Required!

Your Next Big Security Credential: No Prerequisites Needed We know you’ve got the skills, and now, earning the ...

Splunk Observability Cloud's AI Assistant in Action Series: Observability as Code

This is the sixth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to ...

Splunk Answers Content Calendar, July Edition I

Hello Community! Welcome to another month of Community Content Calendar series! For the month of July, we will ...