Deployment Architecture

How can we recover the empty bucket in the var/lib folder after a Splunk system crash?

tlam_splunk
Splunk Employee
Splunk Employee

After a Splunk crash, we are finding that there are a number of emptybucket-hot_v1_xxx in the /var/lib/... folder. Although we can find the new data coming and it can be searched, we are finding that some of the data is missing.

How could we recover the empty bucket ?

highsplunker
Contributor

Thanks a lot! It helped!

0 Karma

tlam_splunk
Splunk Employee
Splunk Employee

After the dirty shutdown, the bucket got corrupted and Splunk marked it for further investigation.

ls -laR emptybucket-hot_v1_xxx

Check that it has the journal.gz and necessary files...

Then do the following
1) Stop Splunk
2) make backup of that bucket
3) rename the bucket back to hot_v1_xxx
4) repair using fsck (and adding --include-hots) (save log output)
5) Start Splunk

Get Updates on the Splunk Community!

Observability | How to Think About Instrumentation Overhead (White Paper)

Novice observability practitioners are often overly obsessed with performance. They might approach ...

Cloud Platform | Get Resiliency in the Cloud Event (Register Now!)

IDC Report: Enterprises Gain Higher Efficiency and Resiliency With Migration to Cloud  Today many enterprises ...

The Great Resilience Quest: 10th Leaderboard Update

The tenth leaderboard update (11.23-12.05) for The Great Resilience Quest is out >> As our brave ...