How to get archived data from frozen buckets in clustered indexers?

[sanglap666]

Hi All,

I want to get  Archived data from Frozen buckets for a certain time frame.
The index which i am trying to fetch is related to windows event logs.
Is their any script available to achive this in clustered environment.

Help in this is much appreciated!

Regards,

Sanglap

[woodcock]

If you didn't setup a frozen policy then your frozen buckets got deleted.
If you did, then it is pretty simple.
Just make sure that you have a thawed directory defined for your index.
Then pick an indexer and just drop the files there and that indexer should start searching them.

[isoutamo]

Hi

Splunk didn't offer any script for that, but you could do your own as many of us have done. Unfortunately I haven't any own as usually did those on customers environments. But with quickly searching I could found at least two which you can use at least as a starting point.

• https://community.splunk.com/t5/Getting-Data-In/thawing-out-multiple-buckets-at-once/td-p/125816
• https://community.splunk.com/t5/Dashboards-Visualizations/How-do-you-rebuild-multiple-buckets-from-f...
• https://community.splunk.com/t5/Splunk-Search/Is-there-a-way-to-restore-archived-data-more-than-1yea...

I haven't test those so check how those are working before using those on production.

r. Ismo.

[yeahnah]

Hi @sanglap666 

You've not described what you frozen index policy does so you should start with sharing that as it could be a bespoke setup.  With saying that, I'm not aware of a thawing script as it really depends on what your frozen policy has done with the data/buckets anyway.  

Here's the the Splunk docs on thawing frozen/archived indexes.

https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Restorearchiveddata

Basically, you can copy the archived data into the thaweddb directory of you indexer peers where Splunk can search it again. 

 $SPLUNK_HOME/var/lib/splunk/<your index>/thaweddb 

 Hope that helps get you started

[sanglap666]

HI @yeahnah ,

This is my policy

[wineventlog]
homePath.maxDataSizeMB = 50000
maxDataSize = auto
maxHotBuckets = 3
repFactor=auto
homePath = $SPLUNK_DB/wineventlog/db
coldPath = /xxxxxxx/splunk/var/lib/wineventlog/colddb
thawedPath = /xxxxx/splunk/var/lib/wineventlog/thaweddb
coldToFrozenDir = /yyyyyy/splunk/var/lib/wineventlog/frozendb
## 60 days in hot
maxHotSpanSecs = 5184000
## 4 months in cold
frozenTimePeriodInSecs = 10368000
tstatsHomePath = volume:_splunk_summaries/wineventlog/datamodel_summary

I see the data in frozen db also i can put it in thawed db as you recommended, but i only want the data in certain time frame say in between jan 2022 to july 2022.

Also i have replicated buckets(rb_...) as well does it needs to be put in the correct indexers based on guid to rebuild it again ?
How can i achive this?

[yeahnah]

Hi @sanglap666 

The index bucket file names use this naming convention db_<newest_time>_<oldest_time>_<localid>_<guid> where the time is in epoch seconds.  Doc ref here...

https://docs.splunk.com/Documentation/Splunk/9.0.4/Indexer/HowSplunkstoresindexes#Bucket_names

Using a Splunk query like this may work, but I'm not 100% sure if state=frozen works or not.

| dbinspect index=wineventlog state=frozen
| eval startDate=strftime(startEpoch,"%A %d %B %Y %H:%M:%S")
| eval endDate=strftime(endEpoch,"%A %d %B %Y %H:%M:%S")
| fields index, path, startDate, endDate, state

 No, you should not need to copy the rb_* buckets and no you should be able to copy the buckets on any indexer and then be able to query the thawed data.