Issue: Cluster Master with multi-site, by mistake wrong retention file was replicated and for the entire indexer in site1 Data got deleted.
Now only peer in site2 has data.As soon as we realized this mistake we have shut down the splunk instance.
What steps can we take to save as much as data possible.
Shutdown all of the indexers and modify indexes.conf and change the retention back to higher value.
Assuming mistake was only on one peer (or one site), you'll lose buckets on that peer. The other buckets will still be around, but marked as frozen.
You need to take the folloing steps to unmark these buckets from “frozen” to “non-frozen”
i)Get a list of frozen buckets using curl commands like