Thawing frozen data is going missing

[alekksi]

Hi all,

We are currently doing backups on frozen buckets before they're removed. Recently we had a request to restore some data which was previously frozen. This was done and the data was available for a while. The following day, the data had gone missing and we had to run the rebuild command on the thawed bucket again to put the data back in.

As per the documentation here, "Data in thaweddb is not subject to the server's index aging scheme (hot > warm> cold > frozen)"

Does anyone have any insight into what might be going on?

Thanks and regards,
Alex

[lycollicott]

Here is a combination of questions and suggestions.
First, your indexer restarted in between the time that you originally thawed the bucket and the time you noticed that you couldn't search it any more?

Take a look in your indexes homePath (for example E:\splunkdb\defaultdb\db) for the file .bucketManifest.

It will contain lines like this:

id,path,"raw_size","event_count","host_count","source_count","sourcetype_count","size_on_disk",modtime,"frozen_in_cluster","origin_site","tsidx_minified"
"your_index~1~D09796FB-2F7E-45EA-8C7C-214C3C6AADE7","rb_1462222380_1458780506_1_D09796FB-2F7E-45EA-8C7C-214C3C6AADE7",2754311,11395,1,34,1,864256,1462235723,0,site2,0
"your_index~3~D09796FB-2F7E-45EA-8C7C-214C3C6AADE7","rb_1462235023_1462222440_3_D09796FB-2F7E-45EA-8C7C-214C3C6AADE7",16502,62,1,2,1,65536,1462235723,0,site2,0
"your_index~4~D09796FB-2F7E-45EA-8C7C-214C3C6AADE7","rb_1462282491_1462235220_4_D09796FB-2F7E-45EA-8C7C-214C3C6AADE7",140152,522,1,2,1,192512,1462549025,0,site2,0

Does the name of your thaweddb folder appear in that file? (I believe that it should be there.)

Is the re-thawed bucket still searchable after the second thawing or have you had to do this multiple times?

[alekksi]

For this restore, I didn't perform any restarts (though I have in the past) as I want to minimise service interruption.
The re-thawed bucket is searchable after re-thawing it again.

I have to replicate the issue to answer the bucket manifest question however, will revert.

Thanks for the suggestions!

[alekksi]

After thawing the same data twice and having it disappear twice, trying to reproduce it now is proving fruitless -- it seems to be working again. While this is great, it's quite frustrating as I can't reproduce the issue.

[lycollicott]

Good luck!

[alekksi]

Try as I might, I can't actually get it to remove the data again. This happened multiple times but now it seems to be working fine. Thank you for your help, but I guess it's all working as expected (worryingly enough).

[lycollicott]

Is the bucket still in the thaweddb directory or do you have to restore it again?

[alekksi]

The bucket remained in thaweddb

[inventsekar]

from a similar issue post - https://answers.splunk.com/answers/337025/after-frozen-data-restore-thawed-data-not-working.html

If you're "thawing" data older than frozenTimePeriodInSecs, it will probably go right back into the frozen directory.

Try creating a new index with defaults and only thaw your previous index into it's thawed directory.

With your original setup, you might find more frozen buckets in $SPLUNK_HOME\Archive\endor (your coldToFrozenDir) as soon as you drop them in thawed and run the rebuild command.

Added after acceptance:
Just to make it clear... hagjos was thawing data back into the same index it was frozen out of. So when he did this, the new thawed data was being detected as older than frozenTimePeriodInSecs and dropping right out of splunk into the frozen directory again.

By creating a new index and thawing into it, he circumvented the immediate freezing of his old data because his new index has a default frozenTimePeriodInSecs of 6 years.

[alekksi]

Thanks for the response, but unfortunately that is not a solution. I have a number of eventtypes set up that simply will not work if I restore to another index. While I, as the administrator, would be well aware that it would be put in another index, the users require it to be as seamless as possible.
If you consider that I have a bucket that is well-used and has a frozen period of, say, a month -- what would then be the point of having a thawed location if it would auto-freeze it? By virtue of it having been frozen once, it will automatically match the criteria of getting re-frozen again -- this doesn't make much sense and sounds like a bug to me.
Thanks again for fetching that answer however.

[lycollicott]

He should only thaw in thaweddb - that is it's intended purpose. You are correct in that undesirable things will happen if it is thawed anywhere else.