index still growing above maxTotalDataSizeMB

Path Finder

We are running out of disk space on one of our indexers on our SAN disk. We adjusted the maxTotalDataSizeMB and restarted but the index is still growing (we expected it to shrink to below maxTotalDataSizeMB). It has been up for about an hour.

10:00 1,376,072
10:15 1,376,137
10:30 1,376,416
10:45 1,377,537
11:00 1,378,630

1,000,000 = max size
1,378,554 = current size
Jun 24, 1999 4:16:40 AM
Nov 30, 2010 11:59:00 P

homePath = /local/splunk/defaultdb/db
coldPath = /san/splunk/defaultdb/colddb
thawedPath = /san/splunk/defaultdb/thaweddb
maxTotalDataSizeMB = 1000000
disabled = false
Tags (1)

Splunk Employee
Splunk Employee

Please log a support case if this has not been already done. Support would need to review the configuration settings for that indexer. In the meanwhile, the following link is a good reference

Splunk Employee
Splunk Employee

Support case was logged and related to the existence of inflight-db buckets.

Inflight-db's are not counted towards the value for maxTotalDatasize of the index.
With Splunk down, if inflight-db buckets remain, they are considered stale and can be deleted.
Once those were removed, in the UI > Manager > Indexes > Current Size was reduced.

An enhancement has been requested for Splunk to look for the existence of these inflight-db buckets at start up and automatically delete these or identify (eg. log) for removal.

0 Karma

Splunk Employee
Splunk Employee

There are other indexes besides the default index - are you sure the current size is just of the default index or of the entire data store?

To verify:

du -ch /san/splunk/defaultdb/colddb
du -ch /local/splunk/defaultdb/db
du -ch /san/splunk/defauldb/thaweddb

Keep in mind that Splunk will never age anything out of the thawedPath.

Also, I don't see any indication that you are controlling the number of warm buckets. By default, Splunk will keep 300 warm buckets in the homePath.

Do you see any buckets with the suffix "inflight" attached to them?

Path Finder

yes, there are other indexes besides default, this is just the one that we are trying to reduce the size of. Current size is received from GUI, so it is of default.

There is data in thaweddb.
182G /defaultdb/thaweddb/
946G /defaultdb/colddb/
215G /defaultdb/db/

The more indexs.conf
maxTotalDataSizeMB = 2000000
frozenTimePeriodInSecs = 7776000
maxDataSize = auto_high_volume
maxWarmDBCount = 20

Yes, there are directories with inflight.
3.6G /san/splunk/defaultdb/colddb/inflight-db_1285372740_1285156778_3940/rawdata

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!