How to Configure maxDataSize for high volume?

[DavidCaputo]

Hi,

I have an index in wich I collect a lot of data, approximately 40 Gb/day.
In the indexes.conf, I guess I've made a mistake and configured :

maxDataSize = auto

Now, it looks like I'm loosing data older than 3 month (roughly) and I guess it's due to this parameter.

In the documentation (I should have read it before !), I can see for maxDataSize : "You should use "auto_high_volume" for high-volume indexes ... A "high volume index" would typically be considered one that gets over 10GB of data per day."

1/ Is it possible to change this parameter for an existing index ?
Obviously, regarding the volume I want to ingest, the "auto_high_volume" is more appropriate
(==> "maxDataSize = auto_high_volume" in the indexes.conf)

2/ Is there any other reason why I am losing data ?

Thanks for your help !
David

[smurf · Accepted solution]

Hi,

maxDataSize tells Splunk how large each bucket can be.

If you are losing old data, you could look for one of these settings:

• maxTotalDataSizeMB
• frozenTimePeriodInSecs
• coldPath.maxDataSizeMB / homePath.maxDataSizeMB

Having any of these would limit the amout of data that is stored in the index. If you are losing everything that is older than 3 months, I would especially look for the attribute frozenTimePeriodInSecs (The number of seconds after which indexed data rolls to frozen.).

[DavidCaputo]

Thanks smurf for your quick answer,

frozenTimePeriodInSecs = 48000000 (~ 18 month, I guess it's enough )

For  coldPath.maxDataSizeMB / homePath.maxDataSizeMB, I can see in the doc :

If this attribute is missing or set to 0, Splunk will not constrain the
  size of homePath.
* Highest legal value is 4294967295
* Defaults to 0.

So, I'm going to try first to set the "maxTotalDataSizeMB" to a larger value than the default one.

[smurf]

I find Monitoring Console good place for debugging Indexes.

Try Monitoring Console -> Indexing -> Indexes and Volumes -> Indexes and Volumes: Instance.

There you have a nice overview of all indexes with their sizes, data age, etc.

[DavidCaputo]

It looks like the maxTotalDataSizeMB solve my problem.

Thanks smurf

David