Deployment Architecture

How to restore frozen data when it looks my buckets have corrupted files?

guimilare
Communicator

Hi all.

I'm running Splunk 6.3.2 in Linux machines.

I have to restore some old data to Splunk.
I've followed the steps described at http://docs.splunk.com/Documentation/Splunk/6.3.2/Indexer/Restorearchiveddata but I'm facing an error that I can't solve.

I've copied the frozen bucket db_1438454243_1438385486_4557 to /var/splunk/defaultdb/thaweddb/
After that, I executed the following command:

/opt/splunk/bin/splunk rebuild /var/splunk/defaultdb/thaweddb/db_1438454243_1438385486_4557 
USAGE: splunk rebuild <bucketPath> [<indexName>] [--ignore-read-error] [--no-log] The <indexName> parameter is ignored if provided. Please see 'splunk fsck' for more options.  This command is just a wrapper for 'splunk fsck'.

Redirecting to 'splunkd fsck' with args:
        repair --one-bucket --include-hots --bucket-path=/var/splunk/defaultdb/thaweddb/db_1438454243_1438385486_4557
--log-to--splunkd-log ERROR JournalSlice - Error reading compressed journal while streaming: bad gzip header, provider=/var/splunk/defaultdb/thaweddb/db_1438454243_1438385486_4557/rawdata/journal.gz ERROR BucketBuilder - Error reading rawdata: Error reading compressed journal while streaming: bad gzip header, provider=/var/splunk/defaultdb/thaweddb/db_1438454243_1438385486_4557/rawdata/journal.gz WARN  Fsck - Repair (entire bucket) idx= bucket='/var/splunk/defaultdb/thaweddb/db_1438454243_1438385486_4557' failed: (entire bucket) Rebuild for bkt='/var/splunk/defaultdb/thaweddb/db_1438454243_1438385486_4557' failed: Error reading rawdata: Error reading compressed journal while streaming: bad gzip header, provider=/var/splunk/defaultdb/thaweddb/db_1438454243_1438385486_4557/rawdata/journal.gz Rebuilding bucket failed

Inside /var/splunk/defaultdb/thaweddb/db_1438454243_1438385486_4557/rawdata there are only 3 files: slicesv2.dat, slicemin.dat and journal.gz.

It looks like the buckets have corrupted files, since the command file in linux five me that the journal.gz file is data.
Is that correct or I'm missing something?

Thank in advance.
Best regards.

0 Karma

Masa
Splunk Employee
Splunk Employee

Yes, sounds like your bucket is corrupted.

How about "gunzip journal.gz" Does this work without error? Most likely this will return error, too.
Unfortunately, if that's the case, there is no tool available recover the bucket.

somesoni2
Revered Legend
0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...