We were doing some testing in a lab environment, with a s2 idx cluster. Apparently a lab user shutdown/rebooted the wrong nodes, resulting in the cluster master and one or more of the 3 cluster members were shutdown by accident in the lab.
After starting the idx cluster boxes and restarting splunk, we have noticed some discrepancies in the data we are searching, e.g. we are not seeing all the data we used to see. There appear to be gaps.
We believe the cache manager was corrupted as the data seems to be complete in the s2 (s3 bucket) but not when searching.
We are looking the proper way to reconnect to the existing smartstore (s3 bucket path) and reset the cache manager to re-register all the contents again.
Unfortunately I am not finding the documentation, which I thought I read months ago...
Related to this, one of our use-cases was to test connecting a standalone indexer to the same s2 to retrieve previous years of logs for forensic reasons, without impacting the primary s2 idx cluster.
You don’t need to do anything else. Indexers will download needed files from S3 when you search. Rolling Restart will also do the same since after restart indexers will mark all local cache as invalid.