Why is "Deployment Client is disabled"? How do I resolve this issue?

[fatsug]

Today I noticed that one of the heavy forwarders in our distributed environment was not calling back to the deployment server, fetching config.

Checking the logs on the HF I noticed:

DC:DeploymentClient [3909 MainThread] - target-broker clause is missing.
DC:DeploymentClient [3909 MainThread] - DeploymentClient explicitly disabled through config.
DS_DC_Common [3909 MainThread] - Deployment Client not initialized.
DS_DC_Common [3909 MainThread] - Loading and initializing Deployment Server...
DeploymentServer [3909 MainThread] - Attempting to reload entire DS; reason='init'
DSManager [3909 MainThread] - No serverclasses configured.
DSManager [3909 MainThread] - Loaded count=0 configured SCs

I tried the  "splunk display deploy-client" telling me that the "Deployment Client is disabled."

I am pretty sure this is why the HF is not phoneing home or fetching new config, though I cannot figure out why?

The "deploymentclient.conf" file is identicall for all our HFs, stored in /etc/apps/xxx/default/deploymentclient.conf

A grep-search for "target-broker" revealed no duplicate/hidden/conflicting files locally generated.

Traffic is allowed as I am able to telnet to DS:8089.

I have tried restarting splunk on the HF with no success, same "DC:DeploymentClient" problems.

Why is this only affecting the one HF and not the others? How can I resolve this issue?

Best regards

// G

[gcusello]

Hi @fatsug,

do you see this HF on your Deployment Server?

I think that you should see it as missing.

in the message is idicated that the HF isn't in any ServerClass.

You should at first check the deploymentclient.conf file on your HF: is it addressing the correct DS IP address?

Then you should see on the DS if that HF is in some ServerClass.

Put special attention to the way to manage deploymentclient.conf file: do you deploy it by DS in a dedicated app or manually?

I usually creata a dedicated TA (called e.g. TA_Forwarders) containing only three  files:

• app.conf
• deploymentclient.conf
• outputs.conf

So I can easily manage them.

Ciao.

Giuseppe

[fatsug]

I was expecting to see it missing, yes. Though I do not see any message regarding the HF at all.

It has been a part of our deployment environment since before I started working with it. There are several succesfully deployed apps and it was part of a number of server classes. Then yesterday I tried pushing new config and noticed it never landed on the HF.

The deploymentclient.conf file is in a dedicated app and it is the same as for every other HF in our environment, so it should be OK. I did try to telnet/curl the exact content of the "Uri" and traffic is allowed.

Is there a way to find out "which" config has shut down the deployment client? As far as I know there should be none.

[gcusello]

Hi @fatsug,

probably it happened something that I exerienced some months ago: for an error you deleted that HF from the serverclass deploying the dedicated TA, so the TA was deleted from that HF and it doesn't take the configuration updates.

In this case you should at first add again the HF to the ServerClass and then manually copy the dedicated TA on the HF and restart it.

In this way your can restore the correct configuration.

Ciao.

Giuseppe

[isoutamo]

One more what you could try on your HF side

# show configs on disk
splunk btool deploymentclient list --debug
# OR show current running config
splunk show config deploymentclient

That will show you where it is configured on your HF side and what are this configuration. In many instructions they told to config DS with cli command and it put those on .../etc/system/local which overrides other places.

r. Ismo 

[fatsug]

So this is the strange part:

$ /opt/splunk/bin/splunk btool deploymentclient list --debug
$ /opt/splunk/bin/splunk show config deploymentclient
Your session is invalid. Please login.
Splunk username: 
Password: 
deploymentclient does not exist

Though the configuration files are present in an dedicated app, and has not been changed 

[isoutamo]

Those basically said that splunk cannot find any deploymentclient configuration files and hasn't load those.

Can you check that file and directory permissions are correct and your splunk user can read those files?

[isoutamo · Accepted solution]

You could try to see when it has connected last time by this query

index=_internal host=<YOUR HF name here> source=*splunkd.log sourcetype=splunkd component IN (DC:DeploymentClient, DS:*, HttpPubSubConnection)

[fatsug]

I managed to pinpoint when that the error occured after the latest change from the deployment server. As there was no error to find, I manually copied the app to the HF and replaced the existing one.

After a reboot the HF now phones home and pulls in changes as expected. I suppose that some error with some file during the last rollout caused the issue and just manually applying the same changes solved the problem.

Thank you for all your help and feedback @isoutamo and @gcusello 

[gcusello]

Hi @fatsug,

good for you, see next time!

Please accept one answer for the other people of Community

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the Contributors 😉

[fatsug]

With some modifications I managed to narrow it down to a restart of the HF on June 1st. After that date the DeploymentClient was explicitly disabled through config.

So a small step forward, now to try to figure out what changed. As far as I can see, nothing with regards to the deployment have changed at all. Though something triggered a restart so something changed.