after OS restart splunk data gone

[gregwilliams]

We upgraded from CentOS 6.2 to 6.3 last night. Upon restart the entire /opt/ directory became corrupt and ended up in /opt/lost+found/ meaning that our entire /opt/splunk/ directory is no longer there. The data is in folders like so:#39855279 #39856144 #39857009 #39857874. Even though the directory names are gibberish, the data appears to be intact. Can this be restored? Has anyone had this happen before, or am I SOL? Before the crash, I had roughly 3 months of data.

Architecture:

OS: Centos 6.3

HD: 4 600GB SAS

RAID card: Dell H700 RAID 10

[kallu · Accepted solution]

I'm afraid there is no generic/easy way to restore your /opt/splunk from lost+found. I would re-install Splunk and any apps you might have had and then try manually identify Splunk data files from lost+found and copy them back to their original locations. Renaming Splunk indexes back to original names can be a challenge though. This can help you finding where your indexes were before the crash. If you are lucky, files can be complete and not corrupted but running Splunk fsck will tell you more how your data is.

Alternative for recovering your data from lost+found is to give thought for how difficult it woud be to re-index (some of) the data you had in Splunk before crash?

[gregwilliams]

Thanks kallu, that helped me think of something else to ask.

[dwaddle]

A good time to restore from backups, assuming they exist...