Hi,
We have spec:d volumes for use in out indexes.conf and we are also (trying) to limit this volumes content with the option of MaxVolumeDataSize .
We are trying to use MaxVolumeDataSize as a "Catch all" limit, rolling the oldest buckets in whatever index they may reside in to our cold storage. (on another volume).
However our volume keeps on growing out of space (100%) and therfore it will stop Splunk from indexing. From what i can see we have some indexes that contain a lot data, but that are not actively used / data comming into. These indexes might not have been used the last 30days(+).
I would have assumed that Splunk, when starting to hit the Volume limit specified in MaxVolumeSize, that even those indexes would have been rolled from warm to cold.
Does not Splunk consider these old indexes as a part of itself anymore? There is nothing on this volume other than splunk indexes so i would have assumed splunk to honour the value of MaxVolumeDataSize.
We are running splunk version 4.3.1, build 119532 on red hat linux 6.1
indexes.conf:
[volume:hot]
path = $SPLUNK_DB
maxVolumeDataSizeMB = 90000
(## 90 000 MB = 90 GB, 10GB less than capacity of disk )
[volume:cold]
path = $SPLUNK_COLDDB
maxVolumeDataSizeMB = 390000
(## 490 000 MB = 390 GB, 10GB less than capacity of disk)
[our_index]
homePath = volume:hot/our_index/db
coldPath = volume:cold/our_index/colddb
thawedPath = $SPLUNK_COLDDB/our_index/thaweddb
Are you certain that all of the indexes that write to $SPLUNK_DB
and $SPLUNK_COLDDB
reference volume:hot
and volume:cold
in their homePath
and coldPath
settings?
As per indexes.conf.spec:
maxVolumeDataSizeMB = <integer>
* Optional.
* If set, this attribute will limit the total cumulative size of all databases
that reside on this volume to the maximum size specified, in MB.
In other words : any index that writes to $SPLUNK_DB
or $SPLUNK_COLDDB
without explicit use of the volumes you defined in homePath
or coldPath
will not be subjected to the maxVolumeDataSizeMB
.
Are you certain that all of the indexes that write to $SPLUNK_DB
and $SPLUNK_COLDDB
reference volume:hot
and volume:cold
in their homePath
and coldPath
settings?
As per indexes.conf.spec:
maxVolumeDataSizeMB = <integer>
* Optional.
* If set, this attribute will limit the total cumulative size of all databases
that reside on this volume to the maximum size specified, in MB.
In other words : any index that writes to $SPLUNK_DB
or $SPLUNK_COLDDB
without explicit use of the volumes you defined in homePath
or coldPath
will not be subjected to the maxVolumeDataSizeMB
.
It seems to be working now 🙂
Thanks for your feedback!
Will mark your answers as the solution.
You should find out exactly what configuration is being applied by running $SPLUNK_HOME/bin/splunk cmd btool indexes list --debug
.
Hi Hexx,
Thanks for your replay.
AS far as i can see all our indexes in our distribueted indexes.conf contains volumes stanza.
However we have some "app" / "addon-conf" containing some indexes.conf files not pointing to the volumes.
Maybe that could be one of the causes for the problem.