topic Re: How to monitor data retention policy and tweak accordingly. in Getting Data In

How to monitor data retention policy and tweak accordingly.

davidwaugh — Thu, 26 Sep 2019 08:31:12 GMT

I've searched but havent yet been able to find the answer.
We have a clustered index setup, and lots of data going into different indexes.

We have the indexes defined with

frozenTimePeriodInSecs

and
maxTotalDataSizeMB

I'd like to produce a dashboard if one doesnt already exist to answer the following questions:

What is the oldest data in each index? eg its 183 days old
How much of the total allotted space is each index using for its hot and cold stores.: eg 98% of Hot and 15% of Cold is being used for this index.
How much of the physical disks have we allocated to indexes. For example if all indexes were full, have we allocated 150% of the physical space available? All indexes sit on a HOT disk and a COLD disk. Eg 98% of hot disk is allocated, 150% of cold disk space is allocated
At the current rate of ingestion, what would the retention be if we used 100% of all allocated space available for the index. eg 360 days
What is limiting our retention - is it our maxTotalDataSizeMB or frozenTimePeriodInSecs for each index.

Thanks for your help.

Here is a screenshot showing a typical index definition that is pushed out to our index cluster.

Re: How to monitor data retention policy and tweak accordingly.

gcusello — Thu, 26 Sep 2019 11:10:58 GMT

Hi davidwaugh,
I think that you should see the Monitoring Console App and check if it solves all your needs.
In addition there's the Index Usage App ( https://splunkbase.splunk.com/app/4086/ ) that could be very useful for your needs.
If there are some need that you cannot solve with them, let me know and surely you'll have the support you need.

Bye.
Giuseppe

Re: How to monitor data retention policy and tweak accordingly.

davidwaugh — Thu, 26 Sep 2019 12:38:05 GMT

Thanks very much. I've just installed Index Usage and have used the Monitoring Console. I think it will take a few days to ingest the data for the dashbaords so will let you know.
Thanks for your help.

Re: How to monitor data retention policy and tweak accordingly.

gcusello — Thu, 26 Sep 2019 13:31:02 GMT

Hi davidwaugh,
if you're (o when you'll be) satisfied by this answer, please accept and/or upvote it.
Bye.
Giuseppe

Re: How to monitor data retention policy and tweak accordingly.

davidwaugh — Fri, 27 Sep 2019 13:36:36 GMT

Thanks Index Usage was the answer. Great app!

Re: How to monitor data retention policy and tweak accordingly.

roseg001 — Tue, 05 May 2020 16:57:10 GMT

please can some one help me splunk retention policy stanza for 80 days