Deployment Architecture

Multi-site cluster , wrong Restrictive retention caused data delete. Anyway to stop splunk from deleting rest of teh data.

sat94541
Communicator

Issue: Cluster Master with multi-site, by mistake wrong retention file was replicated and for the entire indexer in site1 Data got deleted.
Now only peer in site2 has data.As soon as we realized this mistake we have shut down the splunk instance.
What steps can we take to save as much as data possible.

0 Karma

scheng_splunk
Splunk Employee
Splunk Employee
0 Karma

rbal_splunk
Splunk Employee
Splunk Employee

Shutdown all of the indexers and modify indexes.conf and change the retention back to higher value.

Assuming mistake was only on one peer (or one site), you'll lose buckets on that peer. The other buckets will still be around, but marked as frozen.
You need to take the folloing steps to unmark these buckets from “frozen” to “non-frozen”

i)Get a list of frozen buckets using curl commands like

https:// :8089/services/cluster/master/buckets?filter=frozen=true
https://:8089/services/cluster/master/buckets?filter=frozen=true&filter=index=

OR, from the CM you can run

| rest services/cluster/master/buckets?filter=frozen=true&filter=index=ABCD

NOTE: Such rest call from CM can be used as long as there’s not a million buckets that are frozen

ii)Now for each of these bucket edit “bucket_info” and flip the frozen flag from 1 to “”

For example for bucket $SPLUNK_HOME/var/lib/splunk//db/db_1476118503_1476117359_6_6DC24692-777A-424E-AE9D-5FD23FAC185B/ bucket_info

or from the CM you can run

| rest services/cluster/master/buckets?filter=frozen=true&filter=index=ABCD

NOTE: Such rest call from CM can be used as long as there’s not a million buckets that are frozen

Change from>

"indextime_et","indextime_lt","frozen_in_cluster"
1475514557,"",1

change to>
"indextime_et","indextime_lt","frozen_in_cluster"
1475514557,"",""

Also for index, delete $SPLUNK_HOME/var/lib/splunk//db/.bucketManifest
After for all the bucket_info is fixed for all index and indexer, you may restart the indexers.

Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...