How come frozenTimePeriodInSecs is not working as expected?

[pkumar9610]

HI Friends,

I have more than 50 Indexes in my Splunk cluster. For a few of the Indexes, the earliest event is showing as 5 Years (checked it by logging in to Indexer UI -> settings -> Indexers) when the frozenTimePeriodInSecs is mentioned as 14days. I am seeing this issue for almost 5 Indexes. Can you please help me how to fix this issue ?

[symp]
homePath = volume:primary/symp/db
coldPath = volume:primary/symp/colddb
thawedPath = $SPLUNK_DB/symp/thaweddb
# 2 weeks index file retension
frozenTimePeriodInSecs=1209600
# creating each bucket of 10gb
maxDataSize=auto_high_volume
maxHotBuckets = 6

[rabbidroid]

As maciep said, its probably a result of a single bucket spanning over a long period, which is usually caused due to bad timestamping.

You can try to find the offending bucket and indexer by searching:

index=symp earliest=-5y latest=-4y 
| eval BucketID = replace(_cd, "(\d+):\d+", "\1") 
| eval IndexTime=_indextime 
| eval TimeStamp=_time 
| eval delta=_indextime-_time 
| eval SplunkIndexer=splunk_server
| eval Raw=_raw 
| convert ctime(IndexTime) 
| convert ctime(TimeStamp) 
| table TimeStamp IndexTime delta Raw BucketID SplunkIndexer

This will give you the delta of the index_time and the event timestamp, as well as the bucket ID and the Splunk server that returned the data.

You can then SSH to the splunk_server and see the actual bucket in the directory for that index.

If you want, you can also search for that bucket, to see if there is actual current data in it, by adding a search command for a BucketID and it corresponding SplunkIndexer that was returned by the previous search before the table line.

Something like this:

index=symp earliest=-14d latest=now
| eval BucketID = replace(_cd, "(\d+):\d+", "\1") 
| eval IndexTime=_indextime 
| eval TimeStamp=_time 
| eval delta=_indextime-_time 
| eval Raw=_raw 
| eval SplunkIndexer=splunk_server
| convert ctime(IndexTime) 
| convert ctime(TimeStamp) 
| search BucketID=<bad_bucket_id> SplunkIndexer=<your_indexer>
| table TimeStamp IndexTime delta Raw BucketID SplunkIndexer

Hope this helps.

[pkumar9610]

Thank you for the info rabbidroid, when I ran the first search query I have got the output with year 2014 - below is the sample. But when I ran the 2nd query, search was canceled remotely or expired.
TimeStamp: 07/06/2014
IndexTime 14:56:32

delta: 10/07/2018 06:06:42

Raw : 134233810
BucketID : free=1404683792 total=6442450944 percentUsed=78.1964379052321 --JvmThreadDetails-- currentThreadsBusy=15 maxThreads=-1 currentThreadCount=25 percentBusy=0.0 --------------------------------------------- 1848

SplunkIndexer : XXX

Does this mean, there data since 2014 in this bucket is specific server which is causing the issue ?
How can we get rid for this old data ?

[rabbidroid]

It looks like the data was ingested on 10/07/2018 06:06:42 with a event time-stamp of 07/06/2014 14:56:32. The data is not necessarily old, it was just never on-boarded correctly.

Your event does not have a timestamp, so you need to DATETIME_CONFIG = CURRENT in props.conf for that sourcetype, otherwise this will keep on happening.

[pkumar9610]

by making this change will the retention period issue will fix and old data will be frozen ?

[rabbidroid]

No, you will have to wait for the data that was already indexed to reach the 14 day retention you set it to.

[maciep]

i believe that setting applies to the age of the bucket. And the bucket age is determined by the latest event in the bucket. So if a bucket has an event from 5 years ago and one one from 10 days ago, it's not going to roll to frozen because the bucket's age is 10 days.

Can you try to find those buckets and see if that's the case? There should be some rest endpoints on the cm/indexers that will give that info. Or if you have access to the backend, a bucket's name contains the epoch of its latest/earliest event.

[pkumar9610]

I have verified in the bucket, they are with naming convention db_1537860261_1537813515_1777_52BD800C-D26A-409B-8C1F-276DC8119963 and last date I can see is Sep 26 00:24, we have 9Indexers - I have randomly checked in couple of them, they are same. I don't see anything older