After a Splunk crash, we are finding that there are a number of emptybucket-hot_v1_xxx in the /var/lib/... folder. Although we can find the new data coming and it can be searched, we are finding that some of the data is missing.
How could we recover the empty bucket ?
After the dirty shutdown, the bucket got corrupted and Splunk marked it for further investigation.
Check that it has the journal.gz and necessary files...
Then do the following
1) Stop Splunk
2) make backup of that bucket
3) rename the bucket back to hotv1xxx
4) repair using fsck (and adding --include-hots) (save log output)
5) Start Splunk