I understand it is possible to use Deployment Server to propagate config changes to a Search Head Pool as documented here:
It is clear that a single search head can be designated as the deployment client. It will then sync config to the pool target repository and restart itself. What is not clear is how the other search heads in the pool detect the change then restart. When specifying the restartSplunkd parameter in serverclass.conf it does not appear to be honored by search heads which are not designated as deployment clients, but are part of the pool.
Is this the expected behavior? It makes sense that it is the expected behavior since the other search heads are not managed by DS. If this is the case, then what is the recommended way to automate a restart for members of the pool which are not deployment clients?
When configuring a deployment client from the CLI, the config (including targetURI) is written to etc/system/local/deploymentclient.conf. We are trying a solution whereby we delete deploymentclient.conf from etc/system/local and package it as part of a deployment server app. This app is then synced to the SHP target repository by the search head designated as the deployment client. The effect should be the other search heads in the pool now become implicit deployment clients. Will report back on whether this worked or not, and if there are any unintended problems.
What is not clear is how the other
search heads in the pool detect the
change then restart.
Your suspicion is correct -- the other search heads in the pool do not restart, though they do detect updates to the confs.
Is this the expected behavior?
Yes, this is expected behavior.
If this is the case, then what is the
recommended way to automate a restart
for members of the pool which are not
deployment clients?
Creating a search head pool where one search head acts as a deployment client only really makes sense when pushing apps that do not require restarts.
If you're using deployment server to push apps that do require restarts, I'd recommend foregoing search head pooling entirely; just make each search head a standalone deployment client.
Thank you, ewoo! We are trying something out which might work, description is posted as an answer.