Archive

after OS restart splunk data gone

gregwilliams
Path Finder

We upgraded from CentOS 6.2 to 6.3 last night. Upon restart the entire /opt/ directory became corrupt and ended up in /opt/lost+found/ meaning that our entire /opt/splunk/ directory is no longer there. The data is in folders like so:#39855279 #39856144 #39857009 #39857874. Even though the directory names are gibberish, the data appears to be intact. Can this be restored? Has anyone had this happen before, or am I SOL? Before the crash, I had roughly 3 months of data.

Architecture:

OS: Centos 6.3

HD: 4 600GB SAS

RAID card: Dell H700 RAID 10

Tags (1)
1 Solution

kallu
Communicator

I'm afraid there is no generic/easy way to restore your /opt/splunk from lost+found. I would re-install Splunk and any apps you might have had and then try manually identify Splunk data files from lost+found and copy them back to their original locations. Renaming Splunk indexes back to original names can be a challenge though. This can help you finding where your indexes were before the crash. If you are lucky, files can be complete and not corrupted but running Splunk fsck will tell you more how your data is.

Alternative for recovering your data from lost+found is to give thought for how difficult it woud be to re-index (some of) the data you had in Splunk before crash?

View solution in original post

kallu
Communicator

I'm afraid there is no generic/easy way to restore your /opt/splunk from lost+found. I would re-install Splunk and any apps you might have had and then try manually identify Splunk data files from lost+found and copy them back to their original locations. Renaming Splunk indexes back to original names can be a challenge though. This can help you finding where your indexes were before the crash. If you are lucky, files can be complete and not corrupted but running Splunk fsck will tell you more how your data is.

Alternative for recovering your data from lost+found is to give thought for how difficult it woud be to re-index (some of) the data you had in Splunk before crash?

View solution in original post

gregwilliams
Path Finder

Thanks kallu, that helped me think of something else to ask.

0 Karma

dwaddle
SplunkTrust
SplunkTrust

A good time to restore from backups, assuming they exist...

.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!