Deployment client not getting all matching deployment apps

[chanfoli]

I have a handful of deployment clients which are not pulling an app which should match the hostname specified in the whitelist, I even tried IP address in whitelist and the client still only pulls a subset of apps which is should deploy. Other clients with similar hostnames work just fine.

Examples from serverclass.conf on DS:

[serverClass:prod_web]
restartSplunkd=true
whitelist.0=ord-*ws*
whitelist.1=ord-*jumpweb*
whitelist.2=pdx-*ws*
whitelist.3=pdx-*jumpweb*
whitelist.4=172.20.18.95
whitelist.5=ord-wl-prod-v6adwws*
blacklist.0=*db*
[serverClass:prod_web:app:ts_prod_apache_inputs]

On incomplete deployment client:

# hostname
pdx-wl-prod-v6adwws02

# ls /opt/splunkforwarder/etc/apps/
custom_deploymentclient   learned  SplunkUniversalForwarder
custom_forwarder_outputs  search   ts_prod_linux_inputs

Note that the apache app is not deployed.

On complete deployment client:

#hostname
pdx-wl-prod-v6adwws01

 # ls /opt/splunkforwarder/etc/apps/
custom_deploymentclient   search                    ts_prod_linux_inputs
custom_forwarder_outputs  SplunkUniversalForwarder
learned                    ts_prod_apache_inputs

apache imputs from prod_web app was deployed 😕

Seems that a random few deployment clients are missing the prod_web app. Reloading restarting deployment server and clients has no impact. The ip address in whitelist.4 value is actually the IP of pdx-wl-prod-v6adwws02 which I added to further troubleshoot. The client still will not pull the app. Any ideas?

[kaufmanm · Accepted solution]

Everything in the configs you've posted looks fine. I'd try setting a clientName in deploymentclient.conf and then using that for whitelist.0 of the serverClass. I'd restart the deployment server and then the forwarder. I don't have a great theory as to what is causing the problem. Paste the entire serverclass.conf up to where that ended and it might help with troubleshooting.

[chanfoli]

While I don't like the solution of defining a clientName on the clients which are not matching up to server classes which look correct on the deployment server, the suggestion offered does work. I even defined client name to exactly what hostname should be matching to, after doing this and restarting splunk on the deployment client, the app is deployed as desired.

Thanks,
Sean

[chanfoli]

antlefebvre: inputs.conf and server.conf contain the same hostname as the hostname command returns. Thanks for the comment though.

[chanfoli]

somesoni2: I see Reload entries every minute for the apps that are deployed on the 02 server, nothing about the missing prod_web app. The servers with the full set of apps show the same events with the addition of "reload" entries for the prod_web app.

[antlefebvre]

hostname from the cli isn't always the same as the hostname of the forwarder. Check your $SPLUNK/etc/system/local/inputs.conf and see what the host= is for that server.

[somesoni2]

See if you get any event pertaining to the issue.

index=_internal source=*splunkd.log depl* host=pdx-wl-prod-v6adwws02