Getting Data In

Splunk not receiving data from Universal Forwarder

wanling
Path Finder

In our splunk deployment, we have about 100 universal forwarders installed on PCs and forward data to a splunk server (port: 998) that will further forward the data to another splunk server. The intermediate splunk server also serves as a deployment server (port: 8089) to deploy apps to the PCs. We tested this on about 20 machines initially and it worked fine. But after we expanded the deployment to the rest of the machines, we noticed the intermediate splunk server is rejecting the connections from the clients, citing SSL error.

ERROR TcpInputFd - SSL_ERROR_SYSCALL ret errno:0

ERROR TcpInputFd - SSL Error = error:00000000:lib(0):func(0):reason(0)

ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0

ERROR TcpInputFd - SSL Error for fd from HOST:x.x.x.x, IP:x.x.x.x, PORT:2933

We looked at the splunk log at one of the PCs. The log clearly said that it attempts to open a non-SSL connection "INFO TcpOutputProc - Initializing connection for non-ssl forwaring to x.x.x.x". We are puzzled that why the error is related to SSL at the receiver end.

Any advice?

Tags (1)
0 Karma

delink
Communicator

The first step would be to look over your inputs.conf (on the UF) and outputs.conf (on the IF)configurations for this setup. If you could post the relative details here, we could help you troubleshoot this further.

0 Karma

wanling
Path Finder

We figured out that the SSL error is related to deployment server. Once we disable the connection to the deployment server at the client end, we no longer see this error at the server end.

However, not all the clients get connected the splunk server. We found this error message at the client end.
"Connect to x.x.x.x:y failed. No connection could be made because the target machine actively refused it."

It seems to suggest that the intermediate splunk server rejects the connection request from client. However, we cannot locate any information that tells me the cause of it. Any advice?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...