Getting Data In

rawdata restore possible?

gregwilliams
Path Finder

I am attempting to recover from a hard crash, through no fault of Splunk's. Is it possible to unzip /rawdata/journal.gz and import that raw data back into a new index? I had multiple indexes, so there are are multiple folders with rawdata/journal.gz in them. Just looking to unzip them all into 1 index.

Tags (2)
0 Karma

jrodman
Splunk Employee
Splunk Employee

Adam has the good answer here, but to be explicit: you cannot uncompress journal.gz and re-compress it sanely. slices.dat and journal.gz must agree and both must exist.

If you somehow encounter a corruption mid-journal.gz and need to perform partial recovery because it's hugely important data and you have no backups (!!) and are willing to spend many hours on the problem, get in touch directly (username) or via support.

Generally Splunk shouldn't be capable of generating such a problem as we don't modify these files, so any such problem would have to be OS or hardware faults.

0 Karma

ahattrell_splun
Splunk Employee
Splunk Employee

Found this question whilst searching for the same answer - thought I'd give the answer now I've worked it out.

In my case I had a journal.gz but had no idea where it came from originally.

i) Create a directory for the bucket:

  • mkdir /tmp/newBucket
  • mkdir /tmp/newBucket/rawdata

ii) Copy in the journal.gz

  • cp journal.gz /tmp/newBucket/rawdata/

iii) Now run rebuild

  • ./splunk rebuild /tmp/newBucket

iv) Determine the bucket name. We can get this from the tsidx files.

  • ls /tmp/newBucket/*.tsidx

Gives

  • 1351216920-1351213200-1612246429890249572.tsidx
  • 1353460020-1351213200-999025018140623716.tsidx

The format is db_max_min_uniqid where

  • max is the highest of the first number in the tsidx file - in this case 1353460020
  • min is the lowest of the second number in the tsidx filename - in this case 1351213200
  • uniqid is a unique integer that is not currently used in my target index. I used 1.

Thus we get:

  • mv /tmp/newBucket /tmp/db_1353460020_1351213200_1

v) Copy this directory into the target indexes thaweddb directory. I created an index called recovered so:

  • mv /tmp/db_1353460020_1351213200_1 /Applications/splunk/var/lib/splunk/recovered/thaweddb/

vi) Run:

  • ./splunk _internal call /data/indexes/main/rebuild-metadata-and-manifests

Wait a short while and then run a search on your new index.

melonman
Motivator

I was trying to follow this step with Splunk6.0.2, but it did not work. Any chance since this question answered?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...