Data is deleted from cold db before reaching the retention period ?

[arunsony]

The environment is standalone and installed splunk on D:drive. For particular index declared the db location in F:drive for hot and warm buckets and the cold db location is I:drive. The retention policy as 90 days .

In the indexes.conf the setting are as below.
[ssh_res_ss]
frozenTimePeriodInSecs = 7776000
maxwarmDBCount = 3
disabled = 0
coldPath = I:\splunk\ssh_res_ss\colddb
HomePath = F:\splunk\ssh_res_ss\db
thawedPath = F:\splunk\ssh_res_ss\thaweddb

[mattymo]

hi arunsony!

There are two factors that will cause buckets to move in splunk. Time and Size.

My guess is that you have hit a SIZE limit, like max index size which is 500GB by default:

maxTotalDataSizeMB = <nonnegative integer>
* The maximum size of an index (in MB).
* If an index grows larger than the maximum size, the oldest data is frozen.
* This parameter only applies to hot, warm, and cold buckets.  It does not
  apply to thawed buckets.
* Highest legal value is 4294967295
* Defaults to 500000.

Or the volume size:

maxVolumeDataSizeMB = <positive integer>
* Optional, ignored for storageType=remote
* If set, this attribute limits the total size of all databases that reside
  on this volume to the maximum size specified, in MB.  Note that this it
  will act only on those indexes which reference this volume, not on the
  total size of the path set in the path attribute of this volume.
* If the size is exceeded, Splunk will remove buckets with the oldest value
  of latest time (for a given bucket) across all indexes in the volume,
  until the volume is below the maximum size.  This is the trim operation.
  Note that this can cause buckets to be chilled [moved to cold] directly
  from a hot DB, if those buckets happen to have the least value of
  latest-time (LT) across all indexes in the volume.
* Highest legal value is 4294967295, lowest legal value is 1.

see http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Configureindexstorage & indexes.conf.spec for more see https://docs.splunk.com/Documentation/Splunk/latest/Admin/Indexesconf

Tuning indexes.conf can be tricky, so to help the exact root cause can you post btool output:

./splunk btool indexes list ssh_res_ss --debug

https://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Usebtooltotroubleshootconfigurat...

There is also very helpful view in the management console (aka DMC) that can show you more about your index and the data behavior:

Settings > Management Console > Indexing > Indexes & Volumes > Index Detail: Instance

Here you will be able to see the full config of the index (similar to the btool output above) and it will show you if you have hit any configured limits causing the data to roll.

[arunsony]

Splunk version is 6.0.7 . We have SOS app but not DMC. Where can we see in the UI about whether it hits the volumeDatasize or TotalDataSize.

[mattymo]

yikes! Gotta update that instance my friend!

You should be able to see maxDataSizeMB under settings > Indexes

As fo volumes, I am not sure I recall where, if any place, exposes that via the gui.

You could try Settings > All Configurations

But your best bet will be the cmd prompt and btool

[arunsony]

rrF1es\Surk’bifl>SPuflk btoo indexes list ssh....res...SS --debug
b: \PrograR Fil es\Splunk\etc\appSVI aunher\local\i ndexcs. conf [ssh.resSSJ
D:\Proqraal Filcs\Splunk\et c\system\default\indeXes. conf assureLJTf8 — false
D:\Progcam FUes\Splunk\etc\SyStem\detàUlt\ifldexcs.C041f blockSiÇJnStZe = O
D:\Program Files\Splunk\etc\systcm\detaUlt\ilbdeXeS.Cohlf blocksiqnatureoItIb
ase — blocksignature
o:\proqram F11es\Sp1unk\etC\SySt€Ifl\defaU1t\ifldeXeS conf bucketRebui1(emorY
hint — auto
D:\Progra. Fi1es\Sp1unk\etC\SyStem\1OCa1\ifldeXe5.C0 coldPath — I:\Splun
k\ssh_res...ss\colddb
D:\Proqram Flles\Splunk\etc\SyStefll\defaUlt\lfldex€5. conf coidToFrozenOtr —
D:\Proqrdm FIles\Splunk\etC\SySteflI\defaUlt\ifl(ieXeS.C0t1f coldToFrozenScript
o:\Prograa Flles\Splunk\etC\SySteII\defaUlt\ifldeXes. conf co.ipressRawdata = t
rue
D:\Program Flles\Splunk\etc\SyStQ.I\defaUlt\ifldeXes.COflf defaultoatabase
ain
o:\Program Flles\Splunk\etc\systeIfl\local\indexes.coflf dlsabled — O
D:\Program FlIes\Sp1uflk\etC\SyStefl\dCfaU1t\ifldeXeS. conf enableonllne8ucketR
epair — true
D:\Program FlieS\Splunk\etC\SyStem\defaUlt\ifldeXeS. conf enableReaitimeSearC
h true
D:\Prooram Flles\Splunk\etc\system\iocal\indexes. conf frozenTimePeriodlnS
ecs — 7776000
D:\Prograi F1les\splunk\etc\system\local\indexes.conf ho.ePath — F:\splun
kssh..ses_ss\db
D: Proqram Files\Spiunk\etc\systelm\default\lfldexes. conf indexThreads — auto
D: \Program Fiies\spiunk\etc\system\default\indexes. conf RaxBIoorn&aclzfillBuc
ketAge — 30d
o:\Proqram FllesSplunk\etc\system\default\lndexes.conf maxsucketSizecache
ntrles — O
o:\Proqrain FilesSplunk\etc\systein\default\lndexes.conf maxconcurrentOptiml
zes — 6
D: Program Fi les\Lsplunk\etc\apps\launcher\iocal \1 ndexes. conf maxoatasi ze auto_
hi 9h_voi urne
D: \Program Files Si unk\etc\system\def ault\indexes. conf maosotBuckets — 3
D:\Prograrn Files \Splunketc\system\defaultindexes.conf maxHotzdlesecs — O
D:\Proçram Files Splunk\etc\system\default\indcxes.conf !axNot5parsecs
76000
D:Program Files splurk\etcsysterm\defau1tjndexes.conf .ae*B S
D: Prograrn Files Sp1unk\etc.systern\defau1tlndexes.conf rax1etaEntr1es — 10
00000
D: Prograin Files spiunketc systern\defaulrlndexes.conf r.axRunningprocessGr
oups — 8
D: Program Files sp1unketcsystem\defatlt.1nd»ces.conf aRunnirgPressGr
oipsiowPr1or1ty — 1.
0: Prograrit Files splunketcsystern\defau1t.indees.conf naxTi’eLInrep1icated
NOACkS — 300
D: Proqra,r Files Splunk’etcsy5tem’defaulr indexes.corf raTir7teunreplicated
dthAc(s — 60

[mattymo]

whats up with that paste? hard to read.

Confirm your maxTotalDataSizeMB and Volume config and see if you have breached those sizes