When you apply new bundle at the cluster master, Master first validates it locally & then sets its latest bundle id to new bundle checksum (if master validation succeeds). Now peers download the latest bundle & validate it individually. If one or more peers report any validation error(s), then cluster master will roll back to old bundle and then all the peers will continue with old bundle. But in this case, master won't issue a rolling restart of peers and we will log below error message in the Master's splunkd.log.
"Cannot continue with rolling restart. Found bundle validation errors." Followed by error messages from peers.
Peers also log their bundle validation errors to their respective splunkd.logs. You will find something like: "Informed bundle error status to the master for bundle id=" in the peer's splunkd.log.
Please look for any validation errors from peers & at peers. I can shed some more light on what might have gone wrong if i get to see Master's splunkd.log & slave's splunkd.log (where validation failed, if we confirm thats what happened). Hope this helps.
... View more