Universal Forwarder can't connect to Deployment Server

[BryanBerry]

Spent a day on this and have been seeking help in Splunk IRC. Bout to lose it.

Deployment Server states no clients have contacted server. splunkd.log on the Universal Forwarder contains the following:

10-11-2012 15:47:43.879 -0700 WARN  DeploymentClient - Phonehome thread is now started.

10-11-2012 15:47:43.879 -0700 WARN  DeploymentClient - Unable to send handshake message to deployment server. Error status is: not_connected

Conf on forwarder is located at $SPLUNK_HOME/etc/apps/deployment/default.
deploymentclient.conf contains the following:

[deployment-client]

disabled = false

[target-broker:deploymentServer]
targetUri = 10.45.222.191:8089

App.conf in that same directory contains the following:

[install]

state = enabled

On the Deployment Server, $SPLUNK_HOME/etc/system/local/serverclass.conf looks like this:

[global]

[serverClass:all_dev_forwarder]
filterType = whitelist
#filterType = blacklist
repositoryLocation = /opt/splunk/etc/deployment-apps
restartSplunkd = true
whitelist.0 = 10.45.222.*
#blacklist.0 = 10.45.222.190

I've run

./splunk reload deploy-server

on the Deployment Server and

./splunk restart

on the Universal Forwarder multiple times.

I've confirmed that the Universal Forwarder can hit the Deployment Server on 8089.

I'm stumped and have spent a whole day with something I've done a dozen times before. Anyone have any ideas?

[kgrigsby_splunk]

Splunk UF unable to communicate with Deployment client

Problem:
Customer was having issues with splunk forwarder agents not communicating with the splunk deployment server. All the other servers in the same subnet and configuration were able to communicate.

The deploymentclient.conf and all other settings were similar for the entire batch. Customer had issues with 3 servers that were not communicating. Customer tried using the DNS name and also with the IP address of the deployment client server. Customer confirmed that the network connectivity was available for this subnet.

Things to investigate:

• On one of the affected forwarders, put the following channels in debug: DC:DeploymentClient, DC:UpdateServerclassHandler, DC:HandshakeReplyHandler, DC:PhonehomeThread, DS_DC_Common, and HttpPubSubConnection, restart UF and check splunkd.log.

• Also check spulnkd.log on the deployment server. Grep splunkd.log for 'phone' (grep -nri phone splunkd.log) to find out what messages the phonehome thread is generating. The following message indicates an issue: ' 'ex. 48781:10-07-2016 07:42:42.305 -0400 INFO DC:PhonehomeThread - Attempted handshake 20300 times. Will try to re-subscribe to handshake reply'

• Though working and non-working deployment clients were reportedly in the same subnet in this situation (and may be your situation), there is still the possibility that firewalls exist on the local computer itself. Try a curl call from the forwarder to port 8089 of the deployment server by running the following command: 'curl -ku admin:changeme https://servername:8089/services/deployment/server/applications' or 'iptables -L' to confirm whether connectivity is locally blocked though the port appears to be open and listening.

[Gambit22]

Hello, I've been having the same issue as in this question. I am unable to raise an data or get my forwarder to phone home on my Deployment server. I did step 3 of your suggestion from my forwarding server and I got the attached screenshot for a reply. The issue that I'm having is interpreting what this means? It seems that this request from my forwarder is being blocked but both my forwarder and deployment server are currently operating with their firewalls disabled. I don't understand what exactly is occurring here. Is this a credential issue? I don't have any SSL or authentication enabled to the best of my knowledge yet there is something going on that is blocking communication between my forwarder and the deployment. Any suggestions? 

 

 

[BryanBerry · Accepted solution]

Changed the log.cfg on the Deployment Server to log debug messages and restarted Splunk... Hosts began reporting. Seems like Splunk just needed a restart 😕

[BryanBerry]

Noticed the issue with [target-broker:devDeployServer]. Corrected it.