Getting Data In

MaxVolumeDataSize and old indexes

lmyrefelt
Builder

Hi,
We have spec:d volumes for use in out indexes.conf and we are also (trying) to limit this volumes content with the option of MaxVolumeDataSize .

We are trying to use MaxVolumeDataSize as a "Catch all" limit, rolling the oldest buckets in whatever index they may reside in to our cold storage. (on another volume).

However our volume keeps on growing out of space (100%) and therfore it will stop Splunk from indexing. From what i can see we have some indexes that contain a lot data, but that are not actively used / data comming into. These indexes might not have been used the last 30days(+).

I would have assumed that Splunk, when starting to hit the Volume limit specified in MaxVolumeSize, that even those indexes would have been rolled from warm to cold.

Does not Splunk consider these old indexes as a part of itself anymore? There is nothing on this volume other than splunk indexes so i would have assumed splunk to honour the value of MaxVolumeDataSize.

We are running splunk version 4.3.1, build 119532 on red hat linux 6.1

indexes.conf:


[volume:hot]
path = $SPLUNK_DB
maxVolumeDataSizeMB = 90000
(## 90 000 MB = 90 GB, 10GB less than capacity of disk )

[volume:cold]
path = $SPLUNK_COLDDB
maxVolumeDataSizeMB = 390000
(## 490 000 MB = 390 GB, 10GB less than capacity of disk)

[our_index]
homePath = volume:hot/our_index/db
coldPath = volume:cold/our_index/colddb
thawedPath = $SPLUNK_COLDDB/our_index/thaweddb

1 Solution

hexx
Splunk Employee
Splunk Employee

Are you certain that all of the indexes that write to $SPLUNK_DB and $SPLUNK_COLDDB reference volume:hot and volume:cold in their homePath and coldPath settings?

As per indexes.conf.spec:

maxVolumeDataSizeMB = <integer>
* Optional.
* If set, this attribute will limit the total cumulative size of all databases
that reside on this volume to the maximum size specified, in MB.

In other words : any index that writes to $SPLUNK_DB or $SPLUNK_COLDDB without explicit use of the volumes you defined in homePath or coldPath will not be subjected to the maxVolumeDataSizeMB.

View solution in original post

hexx
Splunk Employee
Splunk Employee

Are you certain that all of the indexes that write to $SPLUNK_DB and $SPLUNK_COLDDB reference volume:hot and volume:cold in their homePath and coldPath settings?

As per indexes.conf.spec:

maxVolumeDataSizeMB = <integer>
* Optional.
* If set, this attribute will limit the total cumulative size of all databases
that reside on this volume to the maximum size specified, in MB.

In other words : any index that writes to $SPLUNK_DB or $SPLUNK_COLDDB without explicit use of the volumes you defined in homePath or coldPath will not be subjected to the maxVolumeDataSizeMB.

lmyrefelt
Builder

It seems to be working now 🙂

Thanks for your feedback!
Will mark your answers as the solution.

0 Karma

hexx
Splunk Employee
Splunk Employee

You should find out exactly what configuration is being applied by running $SPLUNK_HOME/bin/splunk cmd btool indexes list --debug.

0 Karma

lmyrefelt
Builder

Hi Hexx,
Thanks for your replay.

AS far as i can see all our indexes in our distribueted indexes.conf contains volumes stanza.

However we have some "app" / "addon-conf" containing some indexes.conf files not pointing to the volumes.
Maybe that could be one of the causes for the problem.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...