Deployment Architecture

How can we recover the empty bucket in the var/lib folder after a Splunk system crash?

Splunk Employee
Splunk Employee

After a Splunk crash, we are finding that there are a number of emptybucket-hot_v1_xxx in the /var/lib/... folder. Although we can find the new data coming and it can be searched, we are finding that some of the data is missing.

How could we recover the empty bucket ?

0 Karma

Splunk Employee
Splunk Employee

After the dirty shutdown, the bucket got corrupted and Splunk marked it for further investigation.

ls -laR emptybucket-hotv1xxx

Check that it has the journal.gz and necessary files...

Then do the following
1) Stop Splunk
2) make backup of that bucket
3) rename the bucket back to hotv1xxx
4) repair using fsck (and adding --include-hots) (save log output)
5) Start Splunk