Its strange something that has happened in our cluster today .
We find all our roles, except a few , missing from the splunk search head .
We did no change but we performed an app upgrade last night where we pushed bundle to the cluster .
Any idea what might have happened that has caused most of the roles to disappear .
And any idea how to recover ?
I believe you have might pushed the bundle to the cluster through the deployer and you have created your roles locally on your search head. If so, then its high chance that your locally done things have been overwritten with the upgrade that you just did. Regarding the recovery, I don't think there's a way unless you have back up.