Splunk Universal Forwarder vs Deployment Server troubleshooting

[clhall1]

I'm having some issues getting Universal Forwarders to talk to the Deployment Server, and I'm looking for some troubleshooting pointers. Here's the scenario, pretty basic setup.

Splunk Enterprise 7.3 on Ubuntu - also configured as a Deployment Server.
Universal Forwarder installed on Windows workstation with the following command:

msiexec /i splunkforwarder-7.3.2-x64.msi AGREETOLICENSE=yes /quiet RECEIVING_INDEXER="splunkslogs:9997" DEPLOYMENT_SERVER="splunkslogs:8089" 

On the server, when I run netstat, I see established connections on ports 8089 and 9997. But I don't see the client listed under "Forwarder Management" of the Splunk GUI.

Suggestions?

[ivanreis · Accepted solution]

Hi clhall1, please try these troubleshooting steps:
- run a ping command from you deployment server to UF client server or vice-versa.
- Check if windows server is able to reach the deployment server by resolution name, if not, I recommend to use the ipaddress for the deployment server instead server name(run nslookup splunkslogs from windows cli command)
- Check if there is any firewall enabled from the deployment server to windows server, please note the connection to deployment server is 2way connection, because the UF server should send/receive packages from deployment server.
- telnet from Deployment server to windows server and vice-versa, you can test if 2 way communication is working
- Check the all the deployment client messages from the client
- index=_internal component=DC* host=ufservername | stats count by message
- Check the deployment messages on the deployment server
- index=_internal component=DS* host=deploymenservername | stats count by message
- Verify if there is no blacklist setup on the deployment server that prevent the communication to this server
- Verify if you have only one client setup as deployment server
- splunk cmd btool deploymentclient list --debug

In additional, verify this splunk answers -> https://answers.splunk.com/answers/214707/how-to-troubleshoot-why-deployment-client-wont-pho.html

[clhall1]

Thanks. You put me on the right path to finding the answer.

I was able ping back and forth, I could telnet back and forth. I ran wireshark on the Windows endpoint and saw all connections going through fine; but something just wasn't working.

On a whim, I changed the deploymentclient.conf file to include the IP of the deployment server instead of the hostname; and it worked. So I went back and changed my install command to use the IP only; and again everything worked. I think this has something to do with Windows networking, wherein I was specifying "splunkslogs" as the hostname, but Windows was resolving it as "splunkslogs.local" since it's my local home playground network. This is pure speculation though. (I come from the days of WINS for local name resolution, so I haven't done much digging on this .local thing now that WINS is pretty much gone).

At any rate, thanks!

[ivanreis]

This can be a potential problem with the DNS service. Glad to help. Happy Splunking!