Data is deleted from cold db before reaching the retention period ?

New Member

The environment is standalone and installed splunk on D:drive. For particular index declared the db location in F:drive for hot and warm buckets and the cold db location is I:drive. The retention policy as 90 days .

In the indexes.conf the setting are as below.
frozenTimePeriodInSecs = 7776000
maxwarmDBCount = 3
disabled = 0
coldPath = I:\splunk\ssh_res_ss\colddb
HomePath = F:\splunk\ssh_res_ss\db
thawedPath = F:\splunk\ssh_res_ss\thaweddb

Tags (1)
0 Karma

Splunk Employee
Splunk Employee

hi arunsony!

There are two factors that will cause buckets to move in splunk. Time and Size.

My guess is that you have hit a SIZE limit, like max index size which is 500GB by default:

maxTotalDataSizeMB = <nonnegative integer>
* The maximum size of an index (in MB).
* If an index grows larger than the maximum size, the oldest data is frozen.
* This parameter only applies to hot, warm, and cold buckets.  It does not
  apply to thawed buckets.
* Highest legal value is 4294967295
* Defaults to 500000.

Or the volume size:

maxVolumeDataSizeMB = <positive integer>
* Optional, ignored for storageType=remote
* If set, this attribute limits the total size of all databases that reside
  on this volume to the maximum size specified, in MB.  Note that this it
  will act only on those indexes which reference this volume, not on the
  total size of the path set in the path attribute of this volume.
* If the size is exceeded, Splunk will remove buckets with the oldest value
  of latest time (for a given bucket) across all indexes in the volume,
  until the volume is below the maximum size.  This is the trim operation.
  Note that this can cause buckets to be chilled [moved to cold] directly
  from a hot DB, if those buckets happen to have the least value of
  latest-time (LT) across all indexes in the volume.
* Highest legal value is 4294967295, lowest legal value is 1.

see & indexes.conf.spec for more see

Tuning indexes.conf can be tricky, so to help the exact root cause can you post btool output:

./splunk btool indexes list ssh_res_ss --debug

There is also very helpful view in the management console (aka DMC) that can show you more about your index and the data behavior:

Settings > Management Console > Indexing > Indexes & Volumes > Index Detail: Instance

Here you will be able to see the full config of the index (similar to the btool output above) and it will show you if you have hit any configured limits causing the data to roll.

0 Karma

New Member

Splunk version is 6.0.7 . We have SOS app but not DMC. Where can we see in the UI about whether it hits the volumeDatasize or TotalDataSize.

0 Karma

Splunk Employee
Splunk Employee

yikes! Gotta update that instance my friend!

You should be able to see maxDataSizeMB under settings > Indexes

As fo volumes, I am not sure I recall where, if any place, exposes that via the gui.

You could try Settings > All Configurations

But your best bet will be the cmd prompt and btool

0 Karma

New Member

rrF1es\Surk’bifl>SPuflk btoo indexes list ssh....res...SS --debug
b: \PrograR Fil es\Splunk\etc\appSVI aunher\local\i ndexcs. conf [ssh.resSSJ
D:\Proqraal Filcs\Splunk\et c\system\default\indeXes. conf assureLJTf8 — false
D:\Progcam FUes\Splunk\etc\SyStem\detàUlt\ifldexcs.C041f blockSiÇJnStZe = O
D:\Program Files\Splunk\etc\systcm\detaUlt\ilbdeXeS.Cohlf blocksiqnatureoItIb
ase — blocksignature
o:\proqram F11es\Sp1unk\etC\SySt€Ifl\defaU1t\ifldeXeS conf bucketRebui1(emorY
hint — auto
D:\Progra. Fi1es\Sp1unk\etC\SyStem\1OCa1\ifldeXe5.C0 coldPath — I:\Splun
D:\Proqram Flles\Splunk\etc\SyStefll\defaUlt\lfldex€5. conf coidToFrozenOtr —
D:\Proqrdm FIles\Splunk\etC\SySteflI\defaUlt\ifl(ieXeS.C0t1f coldToFrozenScript
o:\Prograa Flles\Splunk\etC\SySteII\defaUlt\ifldeXes. conf co.ipressRawdata = t
D:\Program Flles\Splunk\etc\SyStQ.I\defaUlt\ifldeXes.COflf defaultoatabase
o:\Program Flles\Splunk\etc\systeIfl\local\indexes.coflf dlsabled — O
D:\Program FlIes\Sp1uflk\etC\SyStefl\dCfaU1t\ifldeXeS. conf enableonllne8ucketR
epair — true
D:\Program FlieS\Splunk\etC\SyStem\defaUlt\ifldeXeS. conf enableReaitimeSearC
h true
D:\Prooram Flles\Splunk\etc\system\iocal\indexes. conf frozenTimePeriodlnS
ecs — 7776000
D:\Prograi F1les\splunk\etc\system\local\indexes.conf ho.ePath — F:\splun
D: Proqram Files\Spiunk\etc\systelm\default\lfldexes. conf indexThreads — auto
D: \Program Fiies\spiunk\etc\system\default\indexes. conf RaxBIoorn&aclzfillBuc
ketAge — 30d
o:\Proqram FllesSplunk\etc\system\default\lndexes.conf maxsucketSizecache
ntrles — O
o:\Proqrain FilesSplunk\etc\systein\default\lndexes.conf maxconcurrentOptiml
zes — 6
D: Program Fi les\Lsplunk\etc\apps\launcher\iocal \1 ndexes. conf maxoatasi ze auto_
hi 9h_voi urne
D: \Program Files Si unk\etc\system\def ault\indexes. conf maosotBuckets — 3
D:\Prograrn Files \Splunketc\system\defaultindexes.conf maxHotzdlesecs — O
D:\Proçram Files Splunk\etc\system\default\indcxes.conf !axNot5parsecs
D:Program Files splurk\etcsysterm\defau1tjndexes.conf .ae*B S
D: Prograrn Files Sp1unk\etc.systern\defau1tlndexes.conf rax1etaEntr1es — 10
D: Prograin Files spiunketc systern\defaulrlndexes.conf r.axRunningprocessGr
oups — 8
D: Program Files sp1unketcsystem\defatlt.1nd»ces.conf aRunnirgPressGr
oipsiowPr1or1ty — 1.
0: Prograrit Files splunketcsystern\defau1t.indees.conf naxTi’eLInrep1icated
NOACkS — 300
D: Proqra,r Files Splunk’etcsy5tem’defaulr indexes.corf raTir7teunreplicated
dthAc(s — 60

0 Karma

Splunk Employee
Splunk Employee

whats up with that paste? hard to read.

Confirm your maxTotalDataSizeMB and Volume config and see if you have breached those sizes

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!