Deploy-Clients phoning home but not Updating

[rmcdougal]

I have recently made a change to my deployment configs and they are slowly trickling out to the deployment clients. I made this change (port number change in outputs.conf) last week and it still has only made it out to 4000 of my 6000 forwarders. I can see the forwarders still sending data in on the old port and I can see them talking with the deployment server but they are not updating to the new configs

Any Ideas?

[sowings]

outputs.conf changes require a restart of Splunkd. If your serverclass.conf doesn't specify restartSplunkd = true for deploying that app, that's one possible cause.

Is it possible that the forwarders believe that they have the latest version of the application, so even if you changed the serverclass.conf to add restartSplunkd = true to the definition, and issued splunk reload deploy-server the clients wouldn't receive the restart instruction, because the app is still up to date. You can jiggle its elbow a bit by making a small cosmetic change to the configs in that app (an extra comment line would be enough), and then push out the new app again.

While not a solution, this search may help:

index=_internal (sourcetype=splunkd PackageDownloadRestHandler) OR (sourcetype=splunkd DeployedApplication Downloaded)
| rex "streams/deployment\?name=(?<ds_stream>.+?):(?<ds_class>.+?):(?<ds_app>.+?)\s"
| rex "ServerClass:\s(?<ds_class>.+?),\sapp\s(?<ds_app>.+?)\s"
| transaction startswith="PackageDownloadRestHandler" endswith="DeployedApplication" ds_class, ds_app