Unable to view thawed data

[BARNEYRUDD]

Hi,
I'm testing thawing of some frozen data and it's not working. I have thawed some previously frozen data and am expecting to see it in the search, but the search result returned is empty.

Some questions:
- Could this a bug (I'm following the recommended method from Splunk admin training)?
- How could I take my investigation further?

Procedure:

1. stop Splunk
2. copy frozen data (bucket) to a thawed directory
3. run rebuild command - Splunk rebuilds (this appears to work as I see the metadata files are created (Sources.data, bloomfilter, Hosts.data etc).
4. start Splunk
5. search (index=itops earliest = -365d). Result: 0 events - No results found. Try expanding the time range.

A few more details:
- Running SE version 7.3
- the data is from 2 weeks ago (I set the data in the index to age out/freeze after two days)
- this is a test platform
- Seeing some weird logs in the internal index that I don't understand:

 7/9/19 5:14:53.226 PM   07-09-2019 17:14:53.226 +0200 INFO  DatabaseDirectoryManager - Getting size on disk: Unable to get size on disk for bucket id=itops~5~8D8C5421-3FB9-4E28-A7DA-D62472398A71 path="C:\Program Files\Splunk\var\lib\splunk\itops\thaweddb\db_1561999955_1558706749_5" (This is usually harmless as we may be racing with a rename in BucketMover or the S2SFileReceiver thread, which should be obvious in log file; the previous WARN message about this path can safely be ignored.) caller=getBucketManifestValues

host = XXXXXX
source = C:\Program Files\Splunk\var\log\splunk\splunkd.log
sourcetype = splunkd

[edoardo_vicendo]

I confirm I had the same issue (using Splunk 8.0.5), after a restore thawed buckets were not searchable.

Checking with the REST API call the flag frozen was equal to 1.

I solved restarting the Master Node, even if the REST API call still  give the frozen flag equal to 1 (but lot more information are showed after the restart), the thawed buckets are now searchable.

[Prakash493]

Here you go the Rest APi URL of cluster master : https://clustermasteruri:8089/services/cluster/master/buckets/~

Here is a ref screenshot , if you see the frozen flag=1 and bucket state is saying searchable means the bucket is not searchable due to this bug where its saying frozen = 1 and if that saying frozen flag 0 and bucket state is searchable means u r good to go ;

[Prakash493]

if your problem is solved please accept the answer to help future peoples.

[BARNEYRUDD]

Hi @Prakash493 I tried the REST URL you suggested but got nothing (empty page with no results) and I think it's because my Splunk deployment where I have the problem is not in a cluster, it's standalone. I tried modifying the URL to find an equivalent for a standalone environment but couldn't find anything. Do you know what the equivalent URL for a standalone environment would be? I'm thinking maybe your problem is specific only to a clustered environment.

[Prakash493]

i been running into same issue that you are in splunk 7.0.2 their was a bug it prevents the data to be searchable i ended up with setting up a standalone indexer thaw the data their , rebuild the bucket and integrate with search head cluster then my data was searchable.

Tip: You check that by using splunk rest api to go to that particular index and if you see the frozen flag set to true means your rebuilding of buckets is not working.

[splunkreal]

Hello, same issue with 7.1.4, should it be fixed? Thanks.

[splunkreal]

Restarted the Cluster master and it works now!

[BARNEYRUDD]

Thanks @Prakash493. Could you give me the exact name of the frozen flag you are referencing? And the method to check for it if my REST URL is wrong?

I looked at my problem again and realised I was getting a warning on the rebuild command, not sure if it's important/significant or not. I'm using a windows laptop with restricted admin privileges, so maybe my issue is to do with that.

WARN  Fsck - Failed to rename tmpDir='C:\Program Files\Splunk\var\lib\splunk\ito
ps\db\db_1561999955_1558706749_5-tmp' to stageDir='C:\Program Files\Splunk\var\l
ib\splunk\<myindex>\thaweddb\db_1561999955_1558706749_5-stage'.Reason='ERROR_ACCESS_
DENIED'. Will try to copy contents

So I tried the thaw on a Linux VM running Splunk and my data thawed correctly and was searchable.

I then interrogated the rest endpoints on the Windows and Linux machine, but I didn't find the "frozen" flag you were talking about. I found references to thaw and frozen but nothing surprising. For example fields that reference frozen for the index in question:

coldToFrozenDir - C:\xxxxxxxxx\frozen_directory
coldToFrozenScript - No value
frozenTimePeriodInSecs - 172800

REST API URL I used:

https://127.0.0.1:8089/servicesNS/nobody/search/data/indexes/<my _index>____      .

[Prakash493]

Refer to below comment as i cannot attach pictures here in this thread.

[lhanich1]

any update on this? have any luck figuring it out?

[BARNEYRUDD]

Hi @lhanich1, no updates, I haven't looked any further into this.