Deployment Architecture

Multi-site cluster , wrong Restrictive retention caused data delete. Anyway to stop splunk from deleting rest of teh data.

sat94541
Communicator

Issue: Cluster Master with multi-site, by mistake wrong retention file was replicated and for the entire indexer in site1 Data got deleted.
Now only peer in site2 has data.As soon as we realized this mistake we have shut down the splunk instance.
What steps can we take to save as much as data possible.

0 Karma

scheng_splunk
Splunk Employee
Splunk Employee
0 Karma

rbal_splunk
Splunk Employee
Splunk Employee

Shutdown all of the indexers and modify indexes.conf and change the retention back to higher value.

Assuming mistake was only on one peer (or one site), you'll lose buckets on that peer. The other buckets will still be around, but marked as frozen.
You need to take the folloing steps to unmark these buckets from “frozen” to “non-frozen”

i)Get a list of frozen buckets using curl commands like

https:// :8089/services/cluster/master/buckets?filter=frozen=true
https://:8089/services/cluster/master/buckets?filter=frozen=true&filter=index=

OR, from the CM you can run

| rest services/cluster/master/buckets?filter=frozen=true&filter=index=ABCD

NOTE: Such rest call from CM can be used as long as there’s not a million buckets that are frozen

ii)Now for each of these bucket edit “bucket_info” and flip the frozen flag from 1 to “”

For example for bucket $SPLUNK_HOME/var/lib/splunk//db/db_1476118503_1476117359_6_6DC24692-777A-424E-AE9D-5FD23FAC185B/ bucket_info

or from the CM you can run

| rest services/cluster/master/buckets?filter=frozen=true&filter=index=ABCD

NOTE: Such rest call from CM can be used as long as there’s not a million buckets that are frozen

Change from>

"indextime_et","indextime_lt","frozen_in_cluster"
1475514557,"",1

change to>
"indextime_et","indextime_lt","frozen_in_cluster"
1475514557,"",""

Also for index, delete $SPLUNK_HOME/var/lib/splunk//db/.bucketManifest
After for all the bucket_info is fixed for all index and indexer, you may restart the indexers.

Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

SignalFlow: What? Why? How?

What is SignalFlow? Splunk Observability Cloud’s analytics engine, SignalFlow, opens up a world of in-depth ...

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...