Getting Data In

How to monitor data retention policy and tweak accordingly.

davidwaugh
Path Finder

I've searched but havent yet been able to find the answer.
We have a clustered index setup, and lots of data going into different indexes.

We have the indexes defined with

frozenTimePeriodInSecs

and
maxTotalDataSizeMB

I'd like to produce a dashboard if one doesnt already exist to answer the following questions:

  1. What is the oldest data in each index? eg its 183 days old
  2. How much of the total allotted space is each index using for its hot and cold stores.: eg 98% of Hot and 15% of Cold is being used for this index.
  3. How much of the physical disks have we allocated to indexes. For example if all indexes were full, have we allocated 150% of the physical space available? All indexes sit on a HOT disk and a COLD disk. Eg 98% of hot disk is allocated, 150% of cold disk space is allocated
  4. At the current rate of ingestion, what would the retention be if we used 100% of all allocated space available for the index. eg 360 days
  5. What is limiting our retention - is it our maxTotalDataSizeMB or frozenTimePeriodInSecs for each index.

Thanks for your help.

Here is a screenshot showing a typical index definition that is pushed out to our index cluster.

alt text

0 Karma
1 Solution

gcusello
Legend

Hi davidwaugh,
I think that you should see the Monitoring Console App and check if it solves all your needs.
In addition there's the Index Usage App ( https://splunkbase.splunk.com/app/4086/ ) that could be very useful for your needs.
If there are some need that you cannot solve with them, let me know and surely you'll have the support you need.

Bye.
Giuseppe

View solution in original post

roseg001
New Member

please can some one help me splunk retention policy stanza for 80 days

0 Karma

gcusello
Legend

Hi davidwaugh,
I think that you should see the Monitoring Console App and check if it solves all your needs.
In addition there's the Index Usage App ( https://splunkbase.splunk.com/app/4086/ ) that could be very useful for your needs.
If there are some need that you cannot solve with them, let me know and surely you'll have the support you need.

Bye.
Giuseppe

davidwaugh
Path Finder

Thanks very much. I've just installed Index Usage and have used the Monitoring Console. I think it will take a few days to ingest the data for the dashbaords so will let you know.
Thanks for your help.

0 Karma

gcusello
Legend

Hi davidwaugh,
if you're (o when you'll be) satisfied by this answer, please accept and/or upvote it.
Bye.
Giuseppe

0 Karma

davidwaugh
Path Finder

Thanks Index Usage was the answer. Great app!

0 Karma
Get Updates on the Splunk Community!

Happy CX Day to our Community Superheroes!

Happy 10th Birthday CX Day!What is CX Day? It’s a global celebration recognizing innovation and success in the ...

Check out This Month’s Brand new Splunk Lantern Articles

Splunk Lantern is a customer success center providing advice from Splunk experts on valuable data insights, ...

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...