topic Re: Forwarder Not Reaching Out To Deployment Server in Installation

Forwarder Not Reaching Out To Deployment Server

fr0z0n3 — Wed, 26 Aug 2020 17:16:04 GMT

I'm having an issue with getting the Universal Splunk forwarder to reach out to the deployment server. I have 12 servers that I've configured all the same way and 8 of them are working properly but for some reason these last 4 will not reach out. It's not a firewall issue as I can telnet to 8089 to the deployment server without issue and all of the servers have an entry in the serverlist.conf file on the deployment server. In each server we are seeing this in the splunk.d logs

08-26-2020 11:18:32.341 -0500 DEBUG DC:DeploymentClient - Creating a DeploymentClient instance 08-26-2020 11:18:32.356 -0500 DEBUG DC:DeploymentClient - Setting : disabled=false 08-26-2020 11:18:32.356 -0500 DEBUG DC:DeploymentClient - Setting : workingDir=c:\Program Files\SplunkUniversalForwarder\var\run 08-26-2020 11:18:32.356 -0500 DEBUG DC:DeploymentClient - Setting : clientName=DDF77B18-237A-4753-B250-BC8D91C28FF4 08-26-2020 11:18:32.356 -0500 DEBUG DC:DeploymentClient - Setting : repositoryLocation=c:\Program Files\SplunkUniversalForwarder\etc\apps 08-26-2020 11:18:32.356 -0500 DEBUG DC:DeploymentClient - Setting : serverRepositoryLocationPolicy=acceptSplunkHome 08-26-2020 11:18:32.356 -0500 DEBUG DC:DeploymentClient - Setting : serverEndpointPolicy=acceptAlways 08-26-2020 11:18:32.356 -0500 DEBUG DC:DeploymentClient - Setting : maxRetries=3 08-26-2020 11:18:32.356 -0500 DEBUG DC:DeploymentClient - Setting : waitInSecsBetweenRetries=60 08-26-2020 11:18:32.356 -0500 DEBUG DC:DeploymentClient - Setting : phoneHomeIntervalInSecs=60 08-26-2020 11:18:32.356 -0500 INFO DC:DeploymentClient - target-broker clause is missing. 08-26-2020 11:18:32.356 -0500 DEBUG DC:DeploymentClient - Setting : endpoint=$deploymentServerUri$/services/streams/deployment?name=$tenantName$:$serverClassName$:$appName$ 08-26-2020 11:18:32.356 -0500 DEBUG DC:DeploymentClient - Setting : reloadDSOnAppInstall=false 08-26-2020 11:18:32.356 -0500 WARN DC:DeploymentClient - DeploymentClient explicitly disabled through config. 08-26-2020 11:18:32.356 -0500 DEBUG DC:DeploymentClient - trace 1 08-26-2020 11:18:32.356 -0500 DEBUG DC:DeploymentClient - trace 2 08-26-2020 11:18:32.356 -0500 DEBUG DC:DeploymentClient - trace 3 08-26-2020 11:18:32.356 -0500 DEBUG DC:DeploymentClient - trace 4 08-26-2020 11:18:32.356 -0500 DEBUG DC:DeploymentClient - trace 5 08-26-2020 11:18:32.356 -0500 DEBUG DC:DeploymentClient - trace 6 08-26-2020 11:18:32.356 -0500 DEBUG DC:DeploymentClient - trace 7 08-26-2020 11:18:32.356 -0500 DEBUG DC:DeploymentClient - trace 8 08-26-2020 11:18:32.356 -0500 INFO DS_DC_Common - Deployment Client not initialized. 08-26-2020 11:18:32.356 -0500 INFO DS_DC_Common - Deployment Server not available on a dedicated forwarder.

Our deployment.conf is in the correct place and it explicity has disabled set to false under the [deployment-client] heading. I've uninstalled and installed the forwarder multiple times and restarted the services on the deployment server. It just somehow thinks the Deployment client is disabled by default in the config on these 4 servers.

Re: Forwarder Not Reaching Out To Deployment Server

isoutamo — Wed, 26 Aug 2020 17:27:03 GMT

what you got with command

splunk btool deploymentclient list --debug

on those UFs?

r. Ismo

Re: Forwarder Not Reaching Out To Deployment Server

fr0z0n3 — Wed, 26 Aug 2020 17:31:00 GMT

Should have probably specified I'm on Windows. It did give me some output but it also told me this

Command error: The subcommand 'deploymentclient' is not valid for command 'btool.exe'.

Re: Forwarder Not Reaching Out To Deployment Server

isoutamo — Wed, 26 Aug 2020 17:34:20 GMT

I saw it already on your logs and this should work also in windows. I haven’t any windows installation on my hands but maybe this helps. https://community.splunk.com/t5/Getting-Data-In/Can-not-see-the-output-of-btool-in-windows/td-p/451638
r. Ismo

Re: Forwarder Not Reaching Out To Deployment Server

fr0z0n3 — Wed, 26 Aug 2020 17:50:26 GMT

It doesn't return anything

C:\Program Files\SplunkUniversalForwarder\bin>splunk.exe btool deploymentclient list --debug C:\Program Files\SplunkUniversalForwarder\bin>

Re: Forwarder Not Reaching Out To Deployment Server

isoutamo — Wed, 26 Aug 2020 18:05:57 GMT

Basically that means that you haven’t configured DS for this client. Can you check this also on UF which works with DS?

https://docs.splunk.com/Documentation/Forwarder/8.0.5/Forwarder/Configuretheuniversalforwarder

there are more information how to add DS to UF. The best solution is that you have separate app which contain that part and which is automatically installed together with UF.

r. Ismo

Re: Forwarder Not Reaching Out To Deployment Server

fr0z0n3 — Wed, 26 Aug 2020 18:14:46 GMT

Thanks. Shouldn't that all come from the deployment.conf file I have setup during startup

[deployment-client] disabled = false sslVersions = tls1.2 sslVerifyServerCert = true sslRootCAPath = $SPLUNK_HOME/etc/apps/<path to cert> [target-broker:deploymentServer] targetUri = <deployment server hostname>:8089

Our deployment looks for that file in $SPLUNK_HOME/etc/apps/<path to deployment.conf>

Re: Forwarder Not Reaching Out To Deployment Server

isoutamo — Wed, 26 Aug 2020 18:21:35 GMT

I think that those should be in deploymentclient.conf which is under .../apps/TA_your_deploymentclient/default directory.
Choose what ever name you want to your TA name.

r. Ismo

Re: Forwarder Not Reaching Out To Deployment Server

fr0z0n3 — Wed, 26 Aug 2020 18:27:35 GMT

Thank you! It was all because of a typo in the filename. I had accidentally renamed it deployment.conf instead of deploymentclient.conf.