Deployment server - live class status

[grijhwani]

I'm watching the status for a given deployment class (Manager » Deployment » Deployment server » Class Status) in real time (or near real time by reloading the page every couple of seconds), and I am seeing puzzling behaviour. After changing the serverclass.conf and reloading the deployment server I would expect to see the numbers start at zero and increment as the end-points check in, and then given a static estate and a static configuration I would expect that number to remain stable once all machines had updated their config. What I am actually seeing is the number in the list fluctuates down as well as up (+/- about 10 machines in a total estate of 106), despite the fact the end points themselves have no changes to download. I'm not sure that the list is ever complete or definitive.

Could someone with in-depth knowledge please re-assure me that this is not indicative of unstable behaviour. (Currently running Splunk 5.0.5 for the deployment server, and 5.0.5 or 4.3.2 for the end points.)

[lguinn2 · Accepted solution]

Pre 6.x, I believe that the class status is computed based on a fixed time range (like 10 minutes). I don't think the numbers are reset when the deployment server is reloaded.

If you want to understand exactly what is happening with your deployment server, I would examine the Splunk logs in the _internal index. Here are a few searches that could provide a starting point:

index=_internal component=deployedapplication OR component=deploymentclient
| sort host _time 
| table host _time component message

index=_internal sourcetype=splunkd component=Metrics group=ds_connections* |
rename ip as deploymentClient mgmt as mgmtPort |
fields deploymentClient mgmtPort utsname dsevent |
table _time deploymentClient mgmtPort utsname dsevent 

index=_internal component=deploymentclient phonehome 
| sort host _time 
| table host _time component message

(Note that the internal log file format changed somewhat between 5.x and 6x, so you may have to tweak the search to get exactly what you want.)

[grijhwani]

On one point I can be definite. A deployment server reload undoubtedly clear's the tables completely. Looking at the deployment status of any given class on the GUI or watching a repeated loop of the general deployment status from the command line will demonstrate this.

But it never occurred to me - until I actually experienced it today that the deployment status of a server might be registered only for a fixed window. I rather expected it would remain static (all other things being equal), until some material change triggered a redeployment.

Thanks for the sample queries.