How do I never freeze data in an index?

pcjunkie — Thu, 31 Jan 2013 15:50:52 GMT

I have one thread of data that we'd like to keep basically forever. Over the past 8 years the log has only grown to 210MB so we definitely do not have a problem with size or space.

I Splunk'd this log into it's own index, not mixed in with any other indexes so I could keep the retention settings different. The index settings are (including the defaults):

system     assureUTF8 = false
system     blockSignSize = 0
system     blockSignatureDatabase = _blocksignature
system     bucketRebuildMemoryHint = auto
system     coldPath = volume:cold1/gud
system     coldPath.maxDataSizeMB = 150000
system     coldToFrozenDir = 
system     coldToFrozenScript = 
system     compressRawdata = true
system     defaultDatabase = main
system     enableOnlineBucketRepair = true
system     enableRealtimeSearch = true
system     frozenTimePeriodInSecs = 188697600
system     homePath = volume:hot1/gud
system     homePath.maxDataSizeMB = 50000
system     indexThreads = auto
system     maxBloomBackfillBucketAge = 30d
system     maxConcurrentOptimizes = 3
system     maxDataSize = auto
system     maxHotBuckets = 3
system     maxHotIdleSecs = 0
system     maxHotSpanSecs = 7776000
system     maxMemMB = 5
system     maxMetaEntries = 1000000
system     maxRunningProcessGroups = 20
system     maxRunningProcessGroupsLowPriority = 1
system     maxTotalDataSizeMB = 500000
system     maxWarmDBCount = 300
system     memPoolMB = auto
system     minRawFileSyncSecs = disable
system     partialServiceMetaPeriod = 0
system     quarantineFutureSecs = 2592000
system     quarantinePastSecs = 77760000
system     rawChunkSizeBytes = 131072
system     rotatePeriodInSecs = 60
system     serviceMetaPeriod = 25
system     suppressBannerList = 
system     sync = 0
system     syncMeta = true
system     thawedPath = $SPLUNK_COLDDB/gud/thaweddb
system     throttleCheckPeriod = 15

The problem is this index is deleting data older than about 120 days. The total size of the index is 3MB right now.
According to the way I read this configuration it should not freeze data until it reaches 50GB (homePath.maxDataSizeMB = 50000) or approximately 5.98 years old (frozenTimePeriodInSecs = 188697600).

So why then is it deleting data from the index so soon?

Re: How do I never freeze data in an index?

yannK — Thu, 31 Jan 2013 16:13:12 GMT

because a bucket roll to frozen when :

the bucket is not hot anymore
AND all the events in the buckets are older than the frozen time policy

This is very common if your buckets are new and small.

FYI a hot bucket roll when it reaches : maxHotSpanSecs, maxHotBuckets, maxDataSize (that depends of the system, and ca go tup to 10GB per bucket)

use the| dbinspect index=myindex to check the state of your buckets.

topic How do I never freeze data in an index? in Getting Data In

How do I never freeze data in an index?

Re: How do I never freeze data in an index?