Index size over 90%

[bnolf]

My indexes on several of my indexers is over 90%. Yes, I could increase the size, but you can only do that so much. I also understand that when the index is full, the data is frozen, and by default when it is frozen it is deleted. I can't have it deleted. Also, I would like to set the intervals between, hot, warm, cold and frozen to be 30 days. Once it is frozen it needs to be kept for 5 years. I would like to set up a nfs mount for all indexers for the frozen bucket. Is this possible? This is getting critical and I can't lose any data. Any suggestions or what I need to do would be greatly appreciated and welcomed.

thanks

[ddrillic]

As a side thing, the following by @MuS describes a nice way to get an alert for it at How to setup an alert if any index size goes over 90%?

It's -

| rest /services/data/indexes | eval perc=(currentDBSizeMB * 100 / maxTotalDataSizeMB ) | table title currentDBSizeMB maxTotalDataSizeMB perc

[lguinn2]

It is certainly possible to do what you want. You might want to take a look at the Configure index storage section of the documentation, and some of the related sections. For each index to be saved, in indexes.conf add this to the stanza

coldToFrozenDir=/directorynameofNFSmount/indexname

Each index must have its own directory for frozen data. That's all you need to do to prevent data loss, besides managing the NFS files... Now, to control the time intervals:

You cannot set an "interval" for hot. Hot simply means "data arriving right now." Hot buckets become warm buckets when they either fill or are rolled over for other reasons (system restart, staleness, etc.) There are typically 3-10 hot buckets.

For warm, you can set a size limit and you can set a maximum number of buckets. If you fill approximately 1 bucket per day, you can set maxWarmDBCount=30. That is as close as you can come to a time limit on warm.

For cold, you can set the overall time limit for the index. Any bucket (even a warm bucket) that is older than the time limit will be "frozen" by moving it to the directory you configured. To set the time limit, calculate how long you want to store the data (90 days = 7776000 seconds) and put this in indexes.conf

frozenTimePeriodInSecs = 7776000

After you make these changes to indexes.conf, you will need to restart Splunk for the settings to take effect. Splunk will immediately begin to move buckets as necessary to meet the new settings.