Splunk UF is not connecting with Deployment Server

[ankit13]

Hii everyone,

I have installed Splunk uf 10.0.5 on windows server 2016. The UF needs to connected to DS whose version is 10.2.0. But the UF is not connecting with DS. I have tried checking whether there is a network issue or port is blocked. But Test-Net connection shows that it successfully connects my DS on 8089 port.  Netstat -ano shows that ports are open. When checked the splunkd.log of uf it shows that error not connected to DS, handshake failed or socket error. I am attaching the splunkd.log image for the reference.

when I checked the splunkd.log of DS its shows socket error while idling. below is the image attached from DS for the reference.

Any idea?

Thanks in advance,

Regards,

Ankit Singh

[tscroggins]

Hi @ankit13,

I recommend Splunk Universal Forwarder 9.3.x on Windows Server 2016. Splunk Universal Forwarder 9.3.13 was just released. Splunk has been inconsistent on whether more recent versions of the forwarder are fully tested and supported on Windows Server 2016. Both products will be end of life soon.

As others have said, make sure your TLS configuration is consistent on both the client (deploymentclient.conf and server.conf) and server (server.conf).

You can verify connectivity using the forwarder's bundled version of OpenSSL.

From an elevated command prompt: 

cd ; & "C:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" cmd openssl s_client -connect <host>:8089 -CAfile "C:\Program Files\SplunkUniversalForwarder\etc\auth\cacert.pem" -tls1_2 -verify_return_error

Replace <host> with your deployment server hostname, FQDN, or IP address. If you use an IP address, you can add the -servername parameter to openssl; set it to the common name or a valid subjectAltName value in the server certificate. If you need to use an IP address, check your name resolution configuration (DNS and/or %SystemRoot%\System32\drivers\etc\hosts).

The "cd ; &" syntax just lets the command run as-is under both PowerShell and cmd.exe.

A command that fully emulates your current Splunk configuration requires a copy of your forwarder's deploymentclient.conf and server.conf (remove any encrypted or plaintext secrets before posting).

On failure, OpenSSL will print useful error messages and help isolate root cause.

[kml_uvce]

The issue might be - TLS/cipher mismatch between UF 10.0.5 and DS 10.2.0. Different minor versions can ship different sslVersions / cipherSuite / ecdhCurves defaults, and a FIPS-on-one-side-only setup fails identically. also check if UF tries to connect to DS ove unecrypted HTTP.

[ankit13]

Hii,

I have installed Splunk uf 10.2.1 but the issue remains the same. Also, i have checked the suing the OpenSSL and it is getting connected. When checked on DS for the logs related to that server it shows socket error.

While i have another windows server 2012 server in the same environment but it is properly working.

[PickleRick]

How about you help us help you?

Check the config (with btool), compare the contents of deploymentclient.conf on both working and non-working UF...

[ankit13]

The deploymentclient.conf of both servers are same.

[isoutamo]

Please add those configurations here.

[PickleRick]

This kind of errors usually happens with mismatched TLS settings. My guess would be that you have TLS enabled on management port on DS (which is default setting), but your UF tries to connect to DS over unencrypted HTTP.