topic Re: after OS restart splunk data gone in Deployment Architecture

after OS restart splunk data gone

gregwilliams — Tue, 14 Aug 2012 15:46:31 GMT

We upgraded from CentOS 6.2 to 6.3 last night. Upon restart the entire /opt/ directory became corrupt and ended up in /opt/lost+found/ meaning that our entire /opt/splunk/ directory is no longer there. The data is in folders like so:#39855279 #39856144 #39857009 #39857874. Even though the directory names are gibberish, the data appears to be intact. Can this be restored? Has anyone had this happen before, or am I SOL? Before the crash, I had roughly 3 months of data.

Architecture:

OS: Centos 6.3

HD: 4 600GB SAS

RAID card: Dell H700 RAID 10

Re: after OS restart splunk data gone

kallu — Wed, 15 Aug 2012 08:04:05 GMT

I'm afraid there is no generic/easy way to restore your /opt/splunk from lost+found. I would re-install Splunk and any apps you might have had and then try manually identify Splunk data files from lost+found and copy them back to their original locations. Renaming Splunk indexes back to original names can be a challenge though. This can help you finding where your indexes were before the crash. If you are lucky, files can be complete and not corrupted but running Splunk fsck will tell you more how your data is.

Alternative for recovering your data from lost+found is to give thought for how difficult it woud be to re-index (some of) the data you had in Splunk before crash?

Re: after OS restart splunk data gone

dwaddle — Wed, 15 Aug 2012 12:42:51 GMT

A good time to restore from backups, assuming they exist...

Re: after OS restart splunk data gone

gregwilliams — Wed, 15 Aug 2012 15:49:15 GMT

Thanks kallu, that helped me think of something else to ask.