Why are our 6.4.1 universal forwarders unable to connect to a new 6.4.1 deployment server on Windows?

[sylbaea]

Hello,

I have a new deployment server (also acting as search head) installed on Windows Server 2012 R2 with version 6.4.1.
I have multiple Universal Forwarders installed on misc Windows OS (2008 R2, 2012, 2012 R2) with version 6.4.1

Deployment Server is enabled (confirmed with Splunk CLI), has a local serverclass.conf, a deployed app (TA_Windows), all looks fine from Splunk Web.

All UF are enabled as deployment client :

deploymentclient.conf

[target-broker:deploymentServer]
targetUri = <FQDN>:8089

Required firewall ports are opened, and I confirm server is listening on tcp/8089.
However, none of the clients are able to handshake with the server (even those on same subnet than server).

Enabling DEBUG log, I see the following on client side :

06-04-2016 07:37:01.962 +0000 DEBUG DC:PhonehomeThread - PhonehomeThread::main top-of-loop, DC state=Initial
06-04-2016 07:37:01.962 +0000 DEBUG DC:PhonehomeThread - Attempting handshake
06-04-2016 07:37:01.962 +0000 DEBUG DC:DeploymentClient - Sending message <handshake/> to tenantService/handshake
06-04-2016 07:37:01.962 +0000 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
06-04-2016 07:37:01.962 +0000 DEBUG DC:PhonehomeThread - Handshake not yet finished; will retry every 12.0sec
06-04-2016 07:37:01.962 +0000 DEBUG DC:PhonehomeThread - Phonehome thread will wait for 12.0sec (1)

On server, nothing very useful...

06-04-2016 07:22:53.222 +0000 DEBUG ClientSessionsManager - After running metrics, |_newClients|=0 |_existingClients|=0
06-04-2016 07:23:24.223 +0000 DEBUG ClientSessionsManager - Before running metrics, |_newClients|=0 |_existingClients|=0
06-04-2016 07:23:24.223 +0000 DEBUG ClientSessionsManager - After running metrics, |_newClients|=0 |_existingClients|=0
06-04-2016 07:23:55.222 +0000 DEBUG ClientSessionsManager - Before running metrics, |_newClients|=0 |_existingClients|=0

Any idea? This is a brand new install. I already did a similar setup in the past and it works without problem. Servers are new, this is the latest Splunk version, but I cannot see any other difference.

Thanks.

[sylbaea · Accepted solution]

for unknown reason, my deployment server was configured to use a custom port... Very likely I made a mistake in my configuration. Problem resolved.

[jmallorquin]

Hi,

I think the problem is with the firewall, have you try open the outgoing in the new server. Maybe the incoming port 8089 is open but the firewall is cutting all outgoing

Hope i help you

[sylbaea]

firewall requirements are properly implemented (see my comment about that in question)... Thanks for your suggestion in any case.