Getting Data In
Highlighted

How to monitor data retention policy and tweak accordingly.

Path Finder

I've searched but havent yet been able to find the answer.
We have a clustered index setup, and lots of data going into different indexes.

We have the indexes defined with

frozenTimePeriodInSecs

and
maxTotalDataSizeMB

I'd like to produce a dashboard if one doesnt already exist to answer the following questions:

  1. What is the oldest data in each index? eg its 183 days old
  2. How much of the total allotted space is each index using for its hot and cold stores.: eg 98% of Hot and 15% of Cold is being used for this index.
  3. How much of the physical disks have we allocated to indexes. For example if all indexes were full, have we allocated 150% of the physical space available? All indexes sit on a HOT disk and a COLD disk. Eg 98% of hot disk is allocated, 150% of cold disk space is allocated
  4. At the current rate of ingestion, what would the retention be if we used 100% of all allocated space available for the index. eg 360 days
  5. What is limiting our retention - is it our maxTotalDataSizeMB or frozenTimePeriodInSecs for each index.

Thanks for your help.

Here is a screenshot showing a typical index definition that is pushed out to our index cluster.

alt text

0 Karma
Highlighted

Re: How to monitor data retention policy and tweak accordingly.

Legend

Hi davidwaugh,
I think that you should see the Monitoring Console App and check if it solves all your needs.
In addition there's the Index Usage App ( https://splunkbase.splunk.com/app/4086/ ) that could be very useful for your needs.
If there are some need that you cannot solve with them, let me know and surely you'll have the support you need.

Bye.
Giuseppe

View solution in original post

Highlighted

Re: How to monitor data retention policy and tweak accordingly.

Path Finder

Thanks very much. I've just installed Index Usage and have used the Monitoring Console. I think it will take a few days to ingest the data for the dashbaords so will let you know.
Thanks for your help.

0 Karma
Highlighted

Re: How to monitor data retention policy and tweak accordingly.

Legend

Hi davidwaugh,
if you're (o when you'll be) satisfied by this answer, please accept and/or upvote it.
Bye.
Giuseppe

0 Karma
Highlighted

Re: How to monitor data retention policy and tweak accordingly.

Path Finder

Thanks Index Usage was the answer. Great app!

0 Karma
Highlighted

Re: How to monitor data retention policy and tweak accordingly.

New Member

please can some one help me splunk retention policy stanza for 80 days

0 Karma