We are running out of disk space on one of our indexers on our SAN disk. We adjusted the maxTotalDataSizeMB and restarted but the index is still growing (we expected it to shrink to below maxTotalDataSizeMB). It has been up for about an hour.
10:00 1,376,072 10:15 1,376,137 10:30 1,376,416 10:45 1,377,537 11:00 1,378,630 main 1,000,000 = max size 1,378,554 = current size 72,972,419,131 Jun 24, 1999 4:16:40 AM Nov 30, 2010 11:59:00 P [main] homePath = /local/splunk/defaultdb/db coldPath = /san/splunk/defaultdb/colddb thawedPath = /san/splunk/defaultdb/thaweddb maxTotalDataSizeMB = 1000000 disabled = false
Please log a support case if this has not been already done. Support would need to review the configuration settings for that indexer. In the meanwhile, the following link is a good reference
Support case was logged and related to the existence of inflight-db buckets.
Inflight-db's are not counted towards the value for maxTotalDatasize of the index.
With Splunk down, if inflight-db buckets remain, they are considered stale and can be deleted.
Once those were removed, in the UI > Manager > Indexes > Current Size was reduced.
An enhancement has been requested for Splunk to look for the existence of these inflight-db buckets at start up and automatically delete these or identify (eg. log) for removal.
There are other indexes besides the default index - are you sure the current size is just of the default index or of the entire data store?
du -ch /san/splunk/defaultdb/colddb du -ch /local/splunk/defaultdb/db du -ch /san/splunk/defauldb/thaweddb
Keep in mind that Splunk will never age anything out of the thawedPath.
Also, I don't see any indication that you are controlling the number of warm buckets. By default, Splunk will keep 300 warm buckets in the homePath.
Do you see any buckets with the suffix "inflight" attached to them?
yes, there are other indexes besides default, this is just the one that we are trying to reduce the size of. Current size is received from GUI, so it is of default.
There is data in thaweddb.
The more indexs.conf
maxTotalDataSizeMB = 2000000
frozenTimePeriodInSecs = 7776000
maxDataSize = auto_high_volume
maxWarmDBCount = 20
Yes, there are directories with inflight.