Getting Data In

How to troubleshoot why deployment client wont phonehome with my current configuration?

PGrantham
Path Finder

So I've been banging my head against the wall trying to get my Splunk Universal Forwarders to at least attempt to phonehome to the deployment server. If anyone could help with this I would greatly appreciate it!

I have three CentOS 7 VMs, and what I'm trying to do is create a test environment with one of those VMs being my indexer/search head, the other my deployment server, and the other a host with the Universal Forwarder installed.

I'm using 6.2.1.

On the deployment server I have the following in serverclass.conf:

[global]
whitelist.0 = *
restartSplunkd = true
stateOnClient = enabled

[serverClass:all_forwarders_outputs]
whitelist.0 = *
[serverClass:all_forwarders_outputs:app:all_forwarders_outputs]

I have all_forwarders_outputs in the $SPLUNK_HOME/etc/deployment-apps/ directory and in it's local/inputs.conf I have:

[tcpout]
defaultGroup = indexer_group1

[tcpout:indexer_group1]
server = 10.20.48.10:9997

I have verified that the deployment server is enabled with:

[root@localhost local]# /opt/splunk/bin/splunk display deploy-server
Deployment Server is enabled.

On the host with the Universal Forwarder I have the following in the $SPLUNK_HOME/etc/system/local/deploymentclient.conf:

[target-broker:deploymentServer]
targetUri = 10.20.48.50:8089

I have also verified that the forwarder is running as a deployment client with:

[root@localhost local]# /opt/splunkforwarder/bin/splunk display deploy-client
Deployment Client is enabled.

I have also checked to make sure 8089 could be reached on 10.20.48.50 with telnet.

With that configuration, when I run tcpdump it seems that the forwarder isn't even trying to reach the deployment server. It's not sending any traffic at all.

I've checked splunkd.log, even after setting the DeploymentClient log-level to DEBUG, and there's NOTHING related to the forwarder being a deploymentclient. None of the "PhoneHomeThread woke up" or "PhoneHomeThread waiting" lines you would expect, and no errors.

I've tried dropping the all_forwarders_outputs app directly into the forwarder to make sure that it could at least reach out to the indexer and that worked fine.

I've restarted both the forwarder and the deployment server several times and tried changing the phoneHomeInterval but to no avail.

Anyone have any ideas or suggestions? Anything I missed?

Any help would be much appreciated!

1 Solution

PGrantham
Path Finder

Just realized I never posted my fix to this problem. For anyone who's curious, here's what it was:

The problem turned out to be that my VMs were actually clones of the original I created. That original VM had the interface "eth3" configured in /etc/sysconfig/network-scripts/ifcfg-eth3, which is the configuration that all the traffic was going through. When I cloned the VMs, that interface configuration did not carry over. Once I configured the network interface on the cloned VMs everything worked fine.

View solution in original post

landen99
Motivator

Firewall is the most common problem. Check the firewall with the command on the client:

telnet ip port

If the DS is on a Windows box, make sure that the Windows firewall is either turned off or not blocking.

0 Karma

PGrantham
Path Finder

Just realized I never posted my fix to this problem. For anyone who's curious, here's what it was:

The problem turned out to be that my VMs were actually clones of the original I created. That original VM had the interface "eth3" configured in /etc/sysconfig/network-scripts/ifcfg-eth3, which is the configuration that all the traffic was going through. When I cloned the VMs, that interface configuration did not carry over. Once I configured the network interface on the cloned VMs everything worked fine.

Gambit22
Explorer

Sorry, I'm having the same problem and I'm just wondering what you changed in your configuration? I'mhaving the same problem as my client is never phoning home, but I don't know what you changed in this file? 

 

0 Karma

ssjabid
Explorer

hi,

what exactly did you configure here as i am having the exact same issue with my VMS?
please let me know

Abid

0 Karma

InkerzBrad
Explorer

I'm experiencing the same problem, the client does not phone home at all, any solution?

0 Karma

HiroshiSatoh
Champion

Do you app is defined?
[serverClass:all_forwarders_outputs:app:all_forwarders_outputs]

[serverClass:all_forwarders_outputs:app:*]

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...