Indexer - Indexes not retaining logs for as long as they should

[dylanmnf]

Hello,

I have an issue with the Indexer not retaining logs for the expected period, and I'm really scratching my head.

This is from local/indexes.conf

I have maxVolumeDataSizeMB configured on the volumes to provide ample storage. 

[volume:hot]
path = /var/splunk/db/hot
maxVolumeDataSizeMB = 250000

[volume:cold]
path = /var/splunk/db/cold
maxVolumeDataSizeMB = 1100000

Lots of disk space free too.

df -h | grep splunk
   250G 119G 132G 48% /var/splunk/db/hot
   1.2T 136G 991G 13% /var/splunk/db/cold

I have various indexes with frozenTimePeriodInSecs configured for around 1 month/3months/1 year.

[main]
homePath = volume:hot/defaultdb/db
coldPath = volume:cold/defaultdb/colddb
thawedPath = $SPLUNK_DB/defaultdb/thaweddb
frozenTimePeriodInSecs = 8640000

[history]
homePath = volume:hot/historydb/db
coldPath = volume:cold/historydb/colddb
thawedPath = $SPLUNK_DB/historydb/thaweddb

[summary]
homePath = volume:hot/summarydb/db
coldPath = volume:cold/summarydb/colddb
thawedPath = $SPLUNK_DB/summarydb/thaweddb

[_internal]
homePath = volume:hot/_internaldb/db
coldPath = volume:cold/_internaldb/colddb
thawedPath = $SPLUNK_DB/_internaldb/thaweddb
frozenTimePeriodInSecs = 7776000

# For version 6.1 and higher
[_introspection]
homePath = volume:hot/_introspection/db
coldPath = volume:cold/_introspection/colddb
thawedPath = $SPLUNK_DB/_introspection/thaweddb
frozenTimePeriodInSecs = 7776000

# For version 6.5 and higher
[_telemetry]
homePath = volume:hot/_telemetry/db
coldPath = volume:cold/_telemetry/colddb
thawedPath = $SPLUNK_DB/_telemetry/thaweddb
frozenTimePeriodInSecs = 7776000

[_audit]
homePath = volume:hot/audit/db
coldPath = volume:cold/audit/colddb
thawedPath = $SPLUNK_DB/audit/thaweddb
frozenTimePeriodInSecs = 7776000

[_metrics]
homePath = volume:hot/metrics/db
coldPath = volume:cold/metrics/colddb
thawedPath = $SPLUNK_DB/metrics/thaweddb
frozenTimePeriodInSecs = 7776000

[_thefishbucket]
homePath = volume:hot/fishbucket/db
coldPath = volume:cold/fishbucket/colddb
thawedPath = $SPLUNK_DB/fishbucket/thaweddb

[Cisco]
homePath = volume:hot/cisco/db
coldPath = volume:cold/cisco/colddb
thawedPath = $SPLUNK_DB/cisco/thaweddb
frozenTimePeriodInSecs = 3456000

[Windows]
homePath = volume:hot/windows/db
coldPath = volume:cold/windows/colddb
thawedPath = $SPLUNK_DB/windows/thaweddb
frozenTimePeriodInSecs = 31536000

[Linux]
homePath = volume:hot/linux/db
coldPath = volume:cold/linux/colddb
thawedPath = $SPLUNK_DB/linux/thaweddb
frozenTimePeriodInSecs = 31536000

[solaris]
homePath = volume:hot/solaris/db
coldPath = volume:cold/solaris/colddb
thawedPath = $SPLUNK_DB/solaris/thaweddb
frozenTimePeriodInSecs = 31536000

[db]
homePath = volume:hot/db/db
coldPath = volume:cold/db/colddb
thawedPath = $SPLUNK_DB/db/thaweddb
frozenTimePeriodInSecs = 8640000

[Antivirus]
homePath = volume:hot/antivirus/db
coldPath = volume:cold/antivirus/colddb
thawedPath = $SPLUNK_DB/antivirus/thaweddb
frozenTimePeriodInSecs = 8640000

[Mail]
homePath = volume:hot/mail/db
coldPath = volume:cold/mail/colddb
thawedPath = $SPLUNK_DB/mail/thaweddb
frozenTimePeriodInSecs = 8640000

[Test]
homePath = volume:hot/test/db
coldPath = volume:cold/test/colddb
thawedPath = $SPLUNK_DB/test/thaweddb
frozenTimePeriodInSecs = 604800

[msexchange]
homePath = volume:hot/msexchange/db
coldPath = volume:cold/msexchange/colddb
thawedPath = $SPLUNK_DB/msexchange/thaweddb
frozenTimePeriodInSecs = 8640000

[perfmon]
homePath = volume:hot/perfmon/db
coldPath = volume:cold/perfmon/colddb
thawedPath = $SPLUNK_DB/perfmon/thaweddb
frozenTimePeriodInSecs = 8640000

[wineventlog]
homePath = volume:hot/wineventlog/db
coldPath = volume:cold/wineventlog/colddb
thawedPath = $SPLUNK_DB/wineventlog/thaweddb
frozenTimePeriodInSecs = 8640000

[msad]
homePath = volume:hot/msad/db
coldPath = volume:cold/msad/colddb
thawedPath = $SPLUNK_DB/msad/thaweddb
frozenTimePeriodInSecs = 8640000

[proxy]
homePath = volume:hot/proxy/db
coldPath = volume:cold/proxy/colddb
thawedPath = $SPLUNK_DB/proxy/thaweddb
frozenTimePeriodInSecs = 8640000

[servicedesk]
homePath = volume:hot/servicedesk/db
coldPath = volume:cold/servicedesk/colddb
thawedPath = $SPLUNK_DB/servicedesk/thaweddb
frozenTimePeriodInSecs = 8640000

[fortigate]
homePath = volume:hot/fortigate/db
coldPath = volume:cold/fortigate/colddb
thawedPath = $SPLUNK_DB/fortigate/thaweddb
frozenTimePeriodInSecs = 8640000

[cloudflare]
homePath = volume:hot/cloudflare/db
coldPath = volume:cold/cloudflare/colddb
thawedPath = $SPLUNK_DB/cloudflare/thaweddb
frozenTimePeriodInSecs = 8640000

[environmental]
homePath = volume:hot/environmental/db
coldPath = volume:cold/environmental/colddb
thawedPath = $SPLUNK_DB/environmental/thaweddb
frozenTimePeriodInSecs = 8640000

[o365]
homePath = volume:hot/o365/db
coldPath = volume:cold/o365/colddb
thawedPath = $SPLUNK_DB/o365/thaweddb
frozenTimePeriodInSecs = 8640000

[vulnmgmt]
homePath = volume:hot/vulnmgmt/db
coldPath = volume:cold/vulnmgmt/colddb
thawedPath = $SPLUNK_DB/vulnmgmt/thaweddb
frozenTimePeriodInSecs = 31536000
homePath.maxDataSizeMB = 500
coldPath.maxDataSizeMB = 2000

[desktopcentral]
homePath = volume:hot/desktopcentral/db
coldPath = volume:cold/desktopcentral/colddb
thawedPath = $SPLUNK_DB/desktopcentral/thaweddb
frozenTimePeriodInSecs = 31536000
homePath.maxDataSizeMB = 500
coldPath.maxDataSizeMB = 2000

[f5]
homePath = volume:hot/f5/db
coldPath = volume:cold/f5/colddb
thawedPath = $SPLUNK_DB/f5/thaweddb
frozenTimePeriodInSecs = 8640000

[misc]
homePath = volume:hot/misc/db
coldPath = volume:cold/misc/colddb
thawedPath = $SPLUNK_DB/misc/thaweddb
frozenTimePeriodInSecs = 8640000

It seems all the indexes are only storing the last 10 days in the colddb.

ls -l /var/splunk/db/cold/linux/colddb/
total 40
drwx--x---. 3 splunk splunk 4096 Jun 12 16:32 db_1591260601_1591138980_1306
drwx--x---. 3 splunk splunk 4096 Jun 13 19:07 db_1591350601_1591259378_1307
drwx--x---. 3 splunk splunk 4096 Jun 15 08:09 db_1591441801_1591311780_1308
drwx--x---. 3 splunk splunk 4096 Jun 16 09:49 db_1591525801_1591440698_1309
drwx--x---. 3 splunk splunk 4096 Jun 17 10:09 db_1591620001_1591513320_1310
drwx--x---. 3 splunk splunk 4096 Jun 18 03:19 db_1591702201_1590591241_1311
drwx--x---. 3 splunk splunk 4096 Jun 19 01:26 db_1591783801_1591657380_1312
drwx--x---. 3 splunk splunk 4096 Jun 19 22:07 db_1591858861_1591743780_1314
drwx--x---. 3 splunk splunk 4096 Jun 20 22:59 db_1591936201_1591857301_1315
drwx--x---. 3 splunk splunk 4096 Jun 22 05:11 db_1592013661_1591916580_1316

Have I missed something? From my understanding in the docs, I only need to configure maxVolumeDataSizeMB to define the storage capacity, and frozenTimePeriodInSecs for how long logs are kept in cold storage until moved to frozen (deleted).

Thanks

[dylanmnf · Accepted solution]

I found the issue.

I did 

$SPLUNK_HOME$/bin/splunk btool indexes list --debug | grep maxVolumeDataSizeMB

 And found there was another app enabled with maxVolumeDataSizeMB set to 140GB on the volume.

I removed that app and the volume size is working as expected.