topic Re: managing per index data retention in Getting Data In

managing per index data retention

ebaileytu — Tue, 29 Sep 2020 10:17:35 GMT

I need to retain data for 6 months with 400 GB of data max in warm and 5 tb in cold. Will the below work? I am confused since when I use the DMC to review index stats I see index usage described as 488.12/488.28. Does that mean that the index will start dropping data at 488.28? Thanks!

[default]

Default for each index. Can be overridden per index based upon the volume of data received by that index.

400GB

homePath.maxDataSizeMB = 409600

5TB

coldPath.maxDataSizeMB = 5120000

Custom Indexes

[ie]
homePath = $SPLUNK_DB/ie/db
coldPath = /shared/splunk/var/lib/splunk/ie/colddb
thawedPath = $SPLUNK_DB/ie/thaweddb
frozenTimePeriodInSecs = 15552000

Re: managing per index data retention

teunlaan — Fri, 22 Jul 2016 08:49:52 GMT

think you have to set maxTotalDataSizeMB = xxxxxxxxx
Youre setting are still the default.

As far as I understand it will start dropping data on whatever limit is fit first.

Re: managing per index data retention

inventsekar — Fri, 22 Jul 2016 09:10:13 GMT

400GB

homePath.maxDataSizeMB = 400000 --- (did you mistype it as "409600" ah?!?!)

Re: managing per index data retention

inventsekar — Fri, 22 Jul 2016 09:26:48 GMT

Indexes consume most of the disk space. If you run out of disk space, the indexer stops indexing.
You can set a minimum free space limit to control how low free disk space falls
before indexing stops. Indexing resumes once space exceeds the minimum.

Note:
. The indexer does not clear any of its disk space with this method. It simply
pauses until more space becomes available.
· Incoming data can be lost while indexing is suspended.

When a volume containing warm buckets reaches its maxVolumeDataSizeMB, it
starts rolling buckets to cold. When a volume containing cold buckets reaches its
maxVolumeDataSizeMB, it starts rolling buckets to frozen. If a volume contains both
warm and cold buckets (which will happen if an index's homePath and coldPath
are both set to the same volume), the oldest bucket will be rolled to frozen.

Re: managing per index data retention

ebaileytu — Fri, 22 Jul 2016 13:00:27 GMT

ok - that makes sense. I need to define the overall size along with the size for home and cold so data is shifted to cold to account for limits in main storage.

Thanks!