Getting Data In

Splunk Universal Forwarder vs Deployment Server troubleshooting

clhall1
Explorer

I'm having some issues getting Universal Forwarders to talk to the Deployment Server, and I'm looking for some troubleshooting pointers. Here's the scenario, pretty basic setup.

Splunk Enterprise 7.3 on Ubuntu - also configured as a Deployment Server.
Universal Forwarder installed on Windows workstation with the following command:

msiexec /i splunkforwarder-7.3.2-x64.msi AGREETOLICENSE=yes /quiet RECEIVING_INDEXER="splunkslogs:9997" DEPLOYMENT_SERVER="splunkslogs:8089" 

On the server, when I run netstat, I see established connections on ports 8089 and 9997. But I don't see the client listed under "Forwarder Management" of the Splunk GUI.

Suggestions?

0 Karma
1 Solution

ivanreis
Builder

Hi clhall1, please try these troubleshooting steps:
- run a ping command from you deployment server to UF client server or vice-versa.
- Check if windows server is able to reach the deployment server by resolution name, if not, I recommend to use the ipaddress for the deployment server instead server name(run nslookup splunkslogs from windows cli command)
- Check if there is any firewall enabled from the deployment server to windows server, please note the connection to deployment server is 2way connection, because the UF server should send/receive packages from deployment server.
- telnet from Deployment server to windows server and vice-versa, you can test if 2 way communication is working
- Check the all the deployment client messages from the client
- index=_internal component=DC* host=ufservername | stats count by message
- Check the deployment messages on the deployment server
- index=_internal component=DS* host=deploymenservername | stats count by message
- Verify if there is no blacklist setup on the deployment server that prevent the communication to this server
- Verify if you have only one client setup as deployment server
- splunk cmd btool deploymentclient list --debug

In additional, verify this splunk answers -> https://answers.splunk.com/answers/214707/how-to-troubleshoot-why-deployment-client-wont-pho.html

View solution in original post

0 Karma

ivanreis
Builder

Hi clhall1, please try these troubleshooting steps:
- run a ping command from you deployment server to UF client server or vice-versa.
- Check if windows server is able to reach the deployment server by resolution name, if not, I recommend to use the ipaddress for the deployment server instead server name(run nslookup splunkslogs from windows cli command)
- Check if there is any firewall enabled from the deployment server to windows server, please note the connection to deployment server is 2way connection, because the UF server should send/receive packages from deployment server.
- telnet from Deployment server to windows server and vice-versa, you can test if 2 way communication is working
- Check the all the deployment client messages from the client
- index=_internal component=DC* host=ufservername | stats count by message
- Check the deployment messages on the deployment server
- index=_internal component=DS* host=deploymenservername | stats count by message
- Verify if there is no blacklist setup on the deployment server that prevent the communication to this server
- Verify if you have only one client setup as deployment server
- splunk cmd btool deploymentclient list --debug

In additional, verify this splunk answers -> https://answers.splunk.com/answers/214707/how-to-troubleshoot-why-deployment-client-wont-pho.html

0 Karma

clhall1
Explorer

Thanks. You put me on the right path to finding the answer.

I was able ping back and forth, I could telnet back and forth. I ran wireshark on the Windows endpoint and saw all connections going through fine; but something just wasn't working.

On a whim, I changed the deploymentclient.conf file to include the IP of the deployment server instead of the hostname; and it worked. So I went back and changed my install command to use the IP only; and again everything worked. I think this has something to do with Windows networking, wherein I was specifying "splunkslogs" as the hostname, but Windows was resolving it as "splunkslogs.local" since it's my local home playground network. This is pure speculation though. (I come from the days of WINS for local name resolution, so I haven't done much digging on this .local thing now that WINS is pretty much gone).

At any rate, thanks!

0 Karma

ivanreis
Builder

This can be a potential problem with the DNS service. Glad to help. Happy Splunking!

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...