topic Re: rawdata restore possible? in Getting Data In

rawdata restore possible?

gregwilliams — Wed, 15 Aug 2012 15:53:49 GMT

I am attempting to recover from a hard crash, through no fault of Splunk's. Is it possible to unzip /rawdata/journal.gz and import that raw data back into a new index? I had multiple indexes, so there are are multiple folders with rawdata/journal.gz in them. Just looking to unzip them all into 1 index.

Re: rawdata restore possible?

ahattrell_splun — Wed, 28 Nov 2012 17:27:40 GMT

Found this question whilst searching for the same answer - thought I'd give the answer now I've worked it out.

In my case I had a journal.gz but had no idea where it came from originally.

i) Create a directory for the bucket:

mkdir /tmp/newBucket
mkdir /tmp/newBucket/rawdata

ii) Copy in the journal.gz

cp journal.gz /tmp/newBucket/rawdata/

iii) Now run rebuild

./splunk rebuild /tmp/newBucket

iv) Determine the bucket name. We can get this from the tsidx files.

ls /tmp/newBucket/*.tsidx

Gives

1351216920-1351213200-1612246429890249572.tsidx
1353460020-1351213200-999025018140623716.tsidx

The format is db_max_min_uniqid where

max is the highest of the first number in the tsidx file - in this case 1353460020
min is the lowest of the second number in the tsidx filename - in this case 1351213200
uniqid is a unique integer that is not currently used in my target index. I used 1.

Thus we get:

mv /tmp/newBucket /tmp/db_1353460020_1351213200_1

v) Copy this directory into the target indexes thaweddb directory. I created an index called recovered so:

mv /tmp/db_1353460020_1351213200_1 /Applications/splunk/var/lib/splunk/recovered/thaweddb/

vi) Run:

./splunk _internal call /data/indexes/main/rebuild-metadata-and-manifests

Wait a short while and then run a search on your new index.

Re: rawdata restore possible?

jrodman — Wed, 24 Jul 2013 00:49:30 GMT

Adam has the good answer here, but to be explicit: you cannot uncompress journal.gz and re-compress it sanely. slices.dat and journal.gz must agree and both must exist.

If you somehow encounter a corruption mid-journal.gz and need to perform partial recovery because it's hugely important data and you have no backups (!!) and are willing to spend many hours on the problem, get in touch directly (username) or via support.

Generally Splunk shouldn't be capable of generating such a problem as we don't modify these files, so any such problem would have to be OS or hardware faults.

Re: rawdata restore possible?

melonman — Wed, 02 Apr 2014 09:49:18 GMT

I was trying to follow this step with Splunk6.0.2, but it did not work. Any chance since this question answered?