topic Re: Unable to view thawed data in Splunk Search

Unable to view thawed data

BARNEYRUDD — Tue, 09 Jul 2019 15:38:34 GMT

Hi,
I'm testing thawing of some frozen data and it's not working. I have thawed some previously frozen data and am expecting to see it in the search, but the search result returned is empty.

Some questions:
- Could this a bug (I'm following the recommended method from Splunk admin training)?
- How could I take my investigation further?

Procedure:

stop Splunk
copy frozen data (bucket) to a thawed directory
run rebuild command - Splunk rebuilds (this appears to work as I see the metadata files are created (Sources.data, bloomfilter, Hosts.data etc).
start Splunk
search (index=itops earliest = -365d). Result: 0 events - No results found. Try expanding the time range.

A few more details:
- Running SE version 7.3
- the data is from 2 weeks ago (I set the data in the index to age out/freeze after two days)
- this is a test platform
- Seeing some weird logs in the internal index that I don't understand:

 7/9/19 5:14:53.226 PM   07-09-2019 17:14:53.226 +0200 INFO  DatabaseDirectoryManager - Getting size on disk: Unable to get size on disk for bucket id=itops~5~8D8C5421-3FB9-4E28-A7DA-D62472398A71 path="C:\Program Files\Splunk\var\lib\splunk\itops\thaweddb\db_1561999955_1558706749_5" (This is usually harmless as we may be racing with a rename in BucketMover or the S2SFileReceiver thread, which should be obvious in log file; the previous WARN message about this path can safely be ignored.) caller=getBucketManifestValues

host = XXXXXX
source = C:\Program Files\Splunk\var\log\splunk\splunkd.log
sourcetype = splunkd

Re: Unable to view thawed data

lhanich1 — Fri, 15 Nov 2019 02:41:56 GMT

any update on this? have any luck figuring it out?

Re: Unable to view thawed data

BARNEYRUDD — Fri, 15 Nov 2019 09:39:14 GMT

Hi @lhanich1, no updates, I haven't looked any further into this.

Re: Unable to view thawed data

Prakash493 — Mon, 18 Nov 2019 22:23:54 GMT

i been running into same issue that you are in splunk 7.0.2 their was a bug it prevents the data to be searchable i ended up with setting up a standalone indexer thaw the data their , rebuild the bucket and integrate with search head cluster then my data was searchable.

Tip: You check that by using splunk rest api to go to that particular index and if you see the frozen flag set to true means your rebuilding of buckets is not working.

Re: Unable to view thawed data

BARNEYRUDD — Tue, 19 Nov 2019 14:35:00 GMT

Thanks @Prakash493. Could you give me the exact name of the frozen flag you are referencing? And the method to check for it if my REST URL is wrong?

I looked at my problem again and realised I was getting a warning on the rebuild command, not sure if it's important/significant or not. I'm using a windows laptop with restricted admin privileges, so maybe my issue is to do with that.

WARN  Fsck - Failed to rename tmpDir='C:\Program Files\Splunk\var\lib\splunk\ito
ps\db\db_1561999955_1558706749_5-tmp' to stageDir='C:\Program Files\Splunk\var\l
ib\splunk\<myindex>\thaweddb\db_1561999955_1558706749_5-stage'.Reason='ERROR_ACCESS_
DENIED'. Will try to copy contents

So I tried the thaw on a Linux VM running Splunk and my data thawed correctly and was searchable.

I then interrogated the rest endpoints on the Windows and Linux machine, but I didn't find the "frozen" flag you were talking about. I found references to thaw and frozen but nothing surprising. For example fields that reference frozen for the index in question:

coldToFrozenDir - C:\xxxxxxxxx\frozen_directory
coldToFrozenScript - No value
frozenTimePeriodInSecs - 172800

REST API URL I used:

https://127.0.0.1:8089/servicesNS/nobody/search/data/indexes/<my _index>____      .

Re: Unable to view thawed data

Prakash493 — Tue, 19 Nov 2019 14:59:56 GMT

Here you go the Rest APi URL of cluster master : https://clustermasteruri:8089/services/cluster/master/buckets/~

Here is a ref screenshot , if you see the frozen flag=1 and bucket state is saying searchable means the bucket is not searchable due to this bug where its saying frozen = 1 and if that saying frozen flag 0 and bucket state is searchable means u r good to go ;

Re: Unable to view thawed data

Prakash493 — Tue, 19 Nov 2019 15:00:24 GMT

Refer to below comment as i cannot attach pictures here in this thread.

Re: Unable to view thawed data

Prakash493 — Thu, 21 Nov 2019 21:29:48 GMT

if your problem is solved please accept the answer to help future peoples.

Re: Unable to view thawed data

BARNEYRUDD — Fri, 22 Nov 2019 08:50:46 GMT

Hi @Prakash493 I tried the REST URL you suggested but got nothing (empty page with no results) and I think it's because my Splunk deployment where I have the problem is not in a cluster, it's standalone. I tried modifying the URL to find an equivalent for a standalone environment but couldn't find anything. Do you know what the equivalent URL for a standalone environment would be? I'm thinking maybe your problem is specific only to a clustered environment.

Re: Unable to view thawed data

splunkreal — Thu, 11 Jun 2020 20:13:01 GMT

Hello, same issue with 7.1.4, should it be fixed? Thanks.

Re: Unable to view thawed data

splunkreal — Thu, 11 Jun 2020 20:54:13 GMT

Restarted the Cluster master and it works now!

Re: Unable to view thawed data

edoardo_vicendo — Thu, 01 Apr 2021 09:00:39 GMT

I confirm I had the same issue (using Splunk 8.0.5), after a restore thawed buckets were not searchable.

Checking with the REST API call the flag frozen was equal to 1.

I solved restarting the Master Node, even if the REST API call still give the frozen flag equal to 1 (but lot more information are showed after the restart), the thawed buckets are now searchable.

Re: Unable to view thawed data

ilgiz — Thu, 02 May 2024 14:25:03 GMT

In clustered Splunk folder names in thaweddb folder should match db_<newest_time><oldest_time>_<bucketid>_<guid> naming convention. Also you can restore data from another indexer, just change the GUID to local (find in etc/instance.cfg). Please note that rb_ prefix should also be renamed to db_