Getting Data In

Indexer - Indexes not retaining logs for as long as they should

dylanmnf
Engager

Hello,

I have an issue with the Indexer not retaining logs for the expected period, and I'm really scratching my head.

 

This is from local/indexes.conf

I have maxVolumeDataSizeMB configured on the volumes to provide ample storage. 

 

[volume:hot]
path = /var/splunk/db/hot
maxVolumeDataSizeMB = 250000

[volume:cold]
path = /var/splunk/db/cold
maxVolumeDataSizeMB = 1100000

 

 

Lots of disk space free too.

 

df -h | grep splunk
   250G 119G 132G 48% /var/splunk/db/hot
   1.2T 136G 991G 13% /var/splunk/db/cold

 

 

I have various indexes with frozenTimePeriodInSecs configured for around 1 month/3months/1 year.

 

[main]
homePath = volume:hot/defaultdb/db
coldPath = volume:cold/defaultdb/colddb
thawedPath = $SPLUNK_DB/defaultdb/thaweddb
frozenTimePeriodInSecs = 8640000

[history]
homePath = volume:hot/historydb/db
coldPath = volume:cold/historydb/colddb
thawedPath = $SPLUNK_DB/historydb/thaweddb

[summary]
homePath = volume:hot/summarydb/db
coldPath = volume:cold/summarydb/colddb
thawedPath = $SPLUNK_DB/summarydb/thaweddb

[_internal]
homePath = volume:hot/_internaldb/db
coldPath = volume:cold/_internaldb/colddb
thawedPath = $SPLUNK_DB/_internaldb/thaweddb
frozenTimePeriodInSecs = 7776000

# For version 6.1 and higher
[_introspection]
homePath = volume:hot/_introspection/db
coldPath = volume:cold/_introspection/colddb
thawedPath = $SPLUNK_DB/_introspection/thaweddb
frozenTimePeriodInSecs = 7776000

# For version 6.5 and higher
[_telemetry]
homePath = volume:hot/_telemetry/db
coldPath = volume:cold/_telemetry/colddb
thawedPath = $SPLUNK_DB/_telemetry/thaweddb
frozenTimePeriodInSecs = 7776000

[_audit]
homePath = volume:hot/audit/db
coldPath = volume:cold/audit/colddb
thawedPath = $SPLUNK_DB/audit/thaweddb
frozenTimePeriodInSecs = 7776000

[_metrics]
homePath = volume:hot/metrics/db
coldPath = volume:cold/metrics/colddb
thawedPath = $SPLUNK_DB/metrics/thaweddb
frozenTimePeriodInSecs = 7776000

[_thefishbucket]
homePath = volume:hot/fishbucket/db
coldPath = volume:cold/fishbucket/colddb
thawedPath = $SPLUNK_DB/fishbucket/thaweddb

[Cisco]
homePath = volume:hot/cisco/db
coldPath = volume:cold/cisco/colddb
thawedPath = $SPLUNK_DB/cisco/thaweddb
frozenTimePeriodInSecs = 3456000

[Windows]
homePath = volume:hot/windows/db
coldPath = volume:cold/windows/colddb
thawedPath = $SPLUNK_DB/windows/thaweddb
frozenTimePeriodInSecs = 31536000

[Linux]
homePath = volume:hot/linux/db
coldPath = volume:cold/linux/colddb
thawedPath = $SPLUNK_DB/linux/thaweddb
frozenTimePeriodInSecs = 31536000

[solaris]
homePath = volume:hot/solaris/db
coldPath = volume:cold/solaris/colddb
thawedPath = $SPLUNK_DB/solaris/thaweddb
frozenTimePeriodInSecs = 31536000

[db]
homePath = volume:hot/db/db
coldPath = volume:cold/db/colddb
thawedPath = $SPLUNK_DB/db/thaweddb
frozenTimePeriodInSecs = 8640000

[Antivirus]
homePath = volume:hot/antivirus/db
coldPath = volume:cold/antivirus/colddb
thawedPath = $SPLUNK_DB/antivirus/thaweddb
frozenTimePeriodInSecs = 8640000

[Mail]
homePath = volume:hot/mail/db
coldPath = volume:cold/mail/colddb
thawedPath = $SPLUNK_DB/mail/thaweddb
frozenTimePeriodInSecs = 8640000

[Test]
homePath = volume:hot/test/db
coldPath = volume:cold/test/colddb
thawedPath = $SPLUNK_DB/test/thaweddb
frozenTimePeriodInSecs = 604800

[msexchange]
homePath = volume:hot/msexchange/db
coldPath = volume:cold/msexchange/colddb
thawedPath = $SPLUNK_DB/msexchange/thaweddb
frozenTimePeriodInSecs = 8640000

[perfmon]
homePath = volume:hot/perfmon/db
coldPath = volume:cold/perfmon/colddb
thawedPath = $SPLUNK_DB/perfmon/thaweddb
frozenTimePeriodInSecs = 8640000

[wineventlog]
homePath = volume:hot/wineventlog/db
coldPath = volume:cold/wineventlog/colddb
thawedPath = $SPLUNK_DB/wineventlog/thaweddb
frozenTimePeriodInSecs = 8640000

[msad]
homePath = volume:hot/msad/db
coldPath = volume:cold/msad/colddb
thawedPath = $SPLUNK_DB/msad/thaweddb
frozenTimePeriodInSecs = 8640000

[proxy]
homePath = volume:hot/proxy/db
coldPath = volume:cold/proxy/colddb
thawedPath = $SPLUNK_DB/proxy/thaweddb
frozenTimePeriodInSecs = 8640000

[servicedesk]
homePath = volume:hot/servicedesk/db
coldPath = volume:cold/servicedesk/colddb
thawedPath = $SPLUNK_DB/servicedesk/thaweddb
frozenTimePeriodInSecs = 8640000

[fortigate]
homePath = volume:hot/fortigate/db
coldPath = volume:cold/fortigate/colddb
thawedPath = $SPLUNK_DB/fortigate/thaweddb
frozenTimePeriodInSecs = 8640000

[cloudflare]
homePath = volume:hot/cloudflare/db
coldPath = volume:cold/cloudflare/colddb
thawedPath = $SPLUNK_DB/cloudflare/thaweddb
frozenTimePeriodInSecs = 8640000

[environmental]
homePath = volume:hot/environmental/db
coldPath = volume:cold/environmental/colddb
thawedPath = $SPLUNK_DB/environmental/thaweddb
frozenTimePeriodInSecs = 8640000

[o365]
homePath = volume:hot/o365/db
coldPath = volume:cold/o365/colddb
thawedPath = $SPLUNK_DB/o365/thaweddb
frozenTimePeriodInSecs = 8640000

[vulnmgmt]
homePath = volume:hot/vulnmgmt/db
coldPath = volume:cold/vulnmgmt/colddb
thawedPath = $SPLUNK_DB/vulnmgmt/thaweddb
frozenTimePeriodInSecs = 31536000
homePath.maxDataSizeMB = 500
coldPath.maxDataSizeMB = 2000

[desktopcentral]
homePath = volume:hot/desktopcentral/db
coldPath = volume:cold/desktopcentral/colddb
thawedPath = $SPLUNK_DB/desktopcentral/thaweddb
frozenTimePeriodInSecs = 31536000
homePath.maxDataSizeMB = 500
coldPath.maxDataSizeMB = 2000

[f5]
homePath = volume:hot/f5/db
coldPath = volume:cold/f5/colddb
thawedPath = $SPLUNK_DB/f5/thaweddb
frozenTimePeriodInSecs = 8640000

[misc]
homePath = volume:hot/misc/db
coldPath = volume:cold/misc/colddb
thawedPath = $SPLUNK_DB/misc/thaweddb
frozenTimePeriodInSecs = 8640000

 

 

It seems all the indexes are only storing the last 10 days in the colddb.

 

ls -l /var/splunk/db/cold/linux/colddb/
total 40
drwx--x---. 3 splunk splunk 4096 Jun 12 16:32 db_1591260601_1591138980_1306
drwx--x---. 3 splunk splunk 4096 Jun 13 19:07 db_1591350601_1591259378_1307
drwx--x---. 3 splunk splunk 4096 Jun 15 08:09 db_1591441801_1591311780_1308
drwx--x---. 3 splunk splunk 4096 Jun 16 09:49 db_1591525801_1591440698_1309
drwx--x---. 3 splunk splunk 4096 Jun 17 10:09 db_1591620001_1591513320_1310
drwx--x---. 3 splunk splunk 4096 Jun 18 03:19 db_1591702201_1590591241_1311
drwx--x---. 3 splunk splunk 4096 Jun 19 01:26 db_1591783801_1591657380_1312
drwx--x---. 3 splunk splunk 4096 Jun 19 22:07 db_1591858861_1591743780_1314
drwx--x---. 3 splunk splunk 4096 Jun 20 22:59 db_1591936201_1591857301_1315
drwx--x---. 3 splunk splunk 4096 Jun 22 05:11 db_1592013661_1591916580_1316

 

 

Have I missed something? From my understanding in the docs, I only need to configure maxVolumeDataSizeMB to define the storage capacity, and frozenTimePeriodInSecs for how long logs are kept in cold storage until moved to frozen (deleted).

Thanks

Labels (1)
0 Karma
1 Solution

dylanmnf
Engager

I found the issue.

I did 

$SPLUNK_HOME$/bin/splunk btool indexes list --debug | grep maxVolumeDataSizeMB

 And found there was another app enabled with maxVolumeDataSizeMB set to 140GB on the volume.

I removed that app and the volume size is working as expected.

View solution in original post

0 Karma

dylanmnf
Engager

I found the issue.

I did 

$SPLUNK_HOME$/bin/splunk btool indexes list --debug | grep maxVolumeDataSizeMB

 And found there was another app enabled with maxVolumeDataSizeMB set to 140GB on the volume.

I removed that app and the volume size is working as expected.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...