Join the Conversation
- Find Answers
- :
- Splunk Administration
- :
- Deployment Architecture
- :
- after OS restart splunk data gone
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
We upgraded from CentOS 6.2 to 6.3 last night. Upon restart the entire /opt/ directory became corrupt and ended up in /opt/lost+found/ meaning that our entire /opt/splunk/ directory is no longer there. The data is in folders like so:#39855279 #39856144 #39857009 #39857874. Even though the directory names are gibberish, the data appears to be intact. Can this be restored? Has anyone had this happen before, or am I SOL? Before the crash, I had roughly 3 months of data.
Architecture:
OS: Centos 6.3
HD: 4 600GB SAS
RAID card: Dell H700 RAID 10
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I'm afraid there is no generic/easy way to restore your /opt/splunk from lost+found. I would re-install Splunk and any apps you might have had and then try manually identify Splunk data files from lost+found and copy them back to their original locations. Renaming Splunk indexes back to original names can be a challenge though. This can help you finding where your indexes were before the crash. If you are lucky, files can be complete and not corrupted but running Splunk fsck will tell you more how your data is.
Alternative for recovering your data from lost+found is to give thought for how difficult it woud be to re-index (some of) the data you had in Splunk before crash?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I'm afraid there is no generic/easy way to restore your /opt/splunk from lost+found. I would re-install Splunk and any apps you might have had and then try manually identify Splunk data files from lost+found and copy them back to their original locations. Renaming Splunk indexes back to original names can be a challenge though. This can help you finding where your indexes were before the crash. If you are lucky, files can be complete and not corrupted but running Splunk fsck will tell you more how your data is.
Alternative for recovering your data from lost+found is to give thought for how difficult it woud be to re-index (some of) the data you had in Splunk before crash?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks kallu, that helped me think of something else to ask.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
A good time to restore from backups, assuming they exist...
Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!
Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.
Index This | What travels the world but is also stuck in place?
Discover New Use Cases: Unlock Greater Value from Your Existing Splunk Data
Continue Your Journey: Join Session 2 of the Data Management and Federation Bootcamp ...
