topic Re: MaxVolumeDataSize and old indexes in Getting Data In

MaxVolumeDataSize and old indexes

lmyrefelt — Mon, 28 Sep 2020 12:00:47 GMT

Hi,
We have spec:d volumes for use in out indexes.conf and we are also (trying) to limit this volumes content with the option of MaxVolumeDataSize .

We are trying to use MaxVolumeDataSize as a "Catch all" limit, rolling the oldest buckets in whatever index they may reside in to our cold storage. (on another volume).

However our volume keeps on growing out of space (100%) and therfore it will stop Splunk from indexing. From what i can see we have some indexes that contain a lot data, but that are not actively used / data comming into. These indexes might not have been used the last 30days(+).

I would have assumed that Splunk, when starting to hit the Volume limit specified in MaxVolumeSize, that even those indexes would have been rolled from warm to cold.

Does not Splunk consider these old indexes as a part of itself anymore? There is nothing on this volume other than splunk indexes so i would have assumed splunk to honour the value of MaxVolumeDataSize.

We are running splunk version 4.3.1, build 119532 on red hat linux 6.1

indexes.conf:

[volume:hot] path = $SPLUNK_DB maxVolumeDataSizeMB = 90000 (## 90 000 MB = 90 GB, 10GB less than capacity of disk )

[volume:cold]
path = $SPLUNK_COLDDB
maxVolumeDataSizeMB = 390000
(## 490 000 MB = 390 GB, 10GB less than capacity of disk)

[our_index]
homePath = volume:hot/our_index/db
coldPath = volume:cold/our_index/colddb
thawedPath = $SPLUNK_COLDDB/our_index/thaweddb

Re: MaxVolumeDataSize and old indexes

hexx — Mon, 02 Jul 2012 21:02:05 GMT

Are you certain that all of the indexes that write to $SPLUNK_DB and $SPLUNK_COLDDB reference volume:hot and volume:cold in their homePath and coldPath settings?

As per indexes.conf.spec:

maxVolumeDataSizeMB = <integer> * Optional. * If set, this attribute will limit the total cumulative size of all databases that reside on this volume to the maximum size specified, in MB.

In other words : any index that writes to $SPLUNK_DB or $SPLUNK_COLDDB without explicit use of the volumes you defined in homePath or coldPath will not be subjected to the maxVolumeDataSizeMB.

Re: MaxVolumeDataSize and old indexes

lmyrefelt — Mon, 09 Jul 2012 07:51:48 GMT

Hi Hexx,
Thanks for your replay.

AS far as i can see all our indexes in our distribueted indexes.conf contains volumes stanza.

However we have some "app" / "addon-conf" containing some indexes.conf files not pointing to the volumes.
Maybe that could be one of the causes for the problem.

Re: MaxVolumeDataSize and old indexes

hexx — Mon, 09 Jul 2012 07:58:43 GMT

You should find out exactly what configuration is being applied by running $SPLUNK_HOME/bin/splunk cmd btool indexes list --debug.

Re: MaxVolumeDataSize and old indexes

lmyrefelt — Mon, 09 Jul 2012 08:32:23 GMT

It seems to be working now 🙂

Thanks for your feedback!
Will mark your answers as the solution.