My issue is solved I manually copied the files from colddb on the recovered node to the colddb location on the production nodes. (enable maintenance-mode, stop splunk on receiving node, copy  files, make sure ownership is correct, start splunk, disable maintenance-mode) The recovered node is currently still in the cluster, because removing it would fill up the remaining indexers a bit too much.. which would lead to data loss again 🙂 When we added some additional RHEL9 peer nodes we will remove the recovered node and life will be good again. Thanks for the tips and clearifying info. ... View more

Did a quick check on the files in the colddb directory. There are 4 different GUID's, which are actually the same as the GUID's for the existing peers. (which makes sense, since I used the original /opt/splunk/etc on the new RHEL9 nodes, which includes the instance.cfg holding the GUID)   $ ls -lrt colddb/ | awk -F_ '{print $5}' | sort -u 10B29386-EAD3-45F6-AFEF-6C5897D7507E 289FAAF8-810C-454E-9CF5-4DEA9C5CA3E7 332E50AC-2BE6-4FFB-96AB-3F7D612A1422 9C46DD6F-782E-4675-8E9B-90CABC42221D   And the current peers :   $ splunk list cluster-peers | grep -v ":" | grep [0-9] 10B29386-EAD3-45F6-AFEF-6C5897D7507E 289FAAF8-810C-454E-9CF5-4DEA9C5CA3E7 332E50AC-2BE6-4FFB-96AB-3F7D612A1422 42C49D52-0A71-4164-91EC-806EAEEEE085 9C46DD6F-782E-4675-8E9B-90CABC42221D   (The 42C49... GUID is from the restored node, holding all the cold buckets) ... View more

Aha... that makes sense ... and explains a lot. I will see if I can restore the cold buckets by renaming the files / setting the correct GUID in the instance.cfg on the restored node. Thanks a lot for pointing me in the right direction.   ... View more

What we did was : Restored 2 old peer nodes from a backup Cloned the master node to setup a shadow cluster and adapted the replication-factor on this clone to 2. This allowed us to make a mini-cluster which is fully balanced (so both restored peer nodes would have all data) I did however noticed that on one of the two recoved nodes the colddb-location remained empty. Placed the shadow-cluster in maintenance and removed one of the peer nodes. Reconfigured this peer to connect to the production cluster Also changed the name in the server.conf and removed the instance.cfg to prevent duplicate peer names and UUID's When I check the "Settings / Indexer Clustering" page on the master it does show the recovered node as well. The "Indexes" tab on this same page shows all indexes are green. But... when I do a search for the earliestTime, the older data which is on the recovered peer is not seen. Only when I add the recovered peer to the distSearch.conf it does see the older events. Also when I remove the recovered peer again from the cluster the older events are also gone again, which indicates those cold buckets were not synced to the production nodes. The buckets are not rolled to frozen, because the frozenTimePeriodInSecs for the index is set to 157248000 (about 5 years) and the data I try to recover is from 2020. And I did just run a dbinspect and it seems not to give any errors on the cold buckets on the restored host. Path is the colddb-path and state is 'cold' as expected   Eventually I would like to remove the recovered peer again from the cluster, since this is still running RHEL7 and it has to be switched off.. So... I am looking for a way to safely get the data on the RHEL9 nodes. And as a side-track I want to get the understanding of how the warm/cold buckets are handled. Because... when they are indeed not replicated it also explains why they were lost in the first place... the RHEL9 nodes were clean installations which replaced the RHEL7 nodes. The rough procedure followed in this migration was : Add an additional "overflow" peer to the cluster and make sure the cluster is synced. Bring down (offline --enforce-counts) one of the RHEL7 nodes and replace it with a clean RHEL9 node. Config from /opt/splunk/etc was taken over from the old RHEL7 node When all nodes were replace, the "overflow" node was removed. So, when cold buckets were not replicated, the were never replicated to the overflow node and eventually were all gone.. ... View more

That did the trick indeed. Thanks a lot. ... View more