How to troubleshoot why deployment client wont phonehome with my current configuration?

[PGrantham]

So I've been banging my head against the wall trying to get my Splunk Universal Forwarders to at least attempt to phonehome to the deployment server. If anyone could help with this I would greatly appreciate it!

I have three CentOS 7 VMs, and what I'm trying to do is create a test environment with one of those VMs being my indexer/search head, the other my deployment server, and the other a host with the Universal Forwarder installed.

I'm using 6.2.1.

On the deployment server I have the following in serverclass.conf:

[global]
whitelist.0 = *
restartSplunkd = true
stateOnClient = enabled

[serverClass:all_forwarders_outputs]
whitelist.0 = *
[serverClass:all_forwarders_outputs:app:all_forwarders_outputs]

I have all_forwarders_outputs in the $SPLUNK_HOME/etc/deployment-apps/ directory and in it's local/inputs.conf I have:

[tcpout]
defaultGroup = indexer_group1

[tcpout:indexer_group1]
server = 10.20.48.10:9997

I have verified that the deployment server is enabled with:

[root@localhost local]# /opt/splunk/bin/splunk display deploy-server
Deployment Server is enabled.

On the host with the Universal Forwarder I have the following in the $SPLUNK_HOME/etc/system/local/deploymentclient.conf:

[target-broker:deploymentServer]
targetUri = 10.20.48.50:8089

I have also verified that the forwarder is running as a deployment client with:

[root@localhost local]# /opt/splunkforwarder/bin/splunk display deploy-client
Deployment Client is enabled.

I have also checked to make sure 8089 could be reached on 10.20.48.50 with telnet.

With that configuration, when I run tcpdump it seems that the forwarder isn't even trying to reach the deployment server. It's not sending any traffic at all.

I've checked splunkd.log, even after setting the DeploymentClient log-level to DEBUG, and there's NOTHING related to the forwarder being a deploymentclient. None of the "PhoneHomeThread woke up" or "PhoneHomeThread waiting" lines you would expect, and no errors.

I've tried dropping the all_forwarders_outputs app directly into the forwarder to make sure that it could at least reach out to the indexer and that worked fine.

I've restarted both the forwarder and the deployment server several times and tried changing the phoneHomeInterval but to no avail.

Anyone have any ideas or suggestions? Anything I missed?

Any help would be much appreciated!

[landen99]

Firewall is the most common problem. Check the firewall with the command on the client:

telnet ip port

If the DS is on a Windows box, make sure that the Windows firewall is either turned off or not blocking.

[PGrantham · Accepted solution]

Just realized I never posted my fix to this problem. For anyone who's curious, here's what it was:

The problem turned out to be that my VMs were actually clones of the original I created. That original VM had the interface "eth3" configured in /etc/sysconfig/network-scripts/ifcfg-eth3, which is the configuration that all the traffic was going through. When I cloned the VMs, that interface configuration did not carry over. Once I configured the network interface on the cloned VMs everything worked fine.

[Gambit22]

Sorry, I'm having the same problem and I'm just wondering what you changed in your configuration? I'mhaving the same problem as my client is never phoning home, but I don't know what you changed in this file? 

[ssjabid]

hi,

what exactly did you configure here as i am having the exact same issue with my VMS?
please let me know

Abid

[InkerzBrad]

I'm experiencing the same problem, the client does not phone home at all, any solution?

[HiroshiSatoh]

Do you app is defined?
[serverClass:all_forwarders_outputs:app:all_forwarders_outputs]
↓
[serverClass:all_forwarders_outputs:app:*]