Getting Data In

How to monitor data retention policy and tweak accordingly.

davidwaugh
Path Finder

I've searched but havent yet been able to find the answer.
We have a clustered index setup, and lots of data going into different indexes.

We have the indexes defined with

frozenTimePeriodInSecs

and
maxTotalDataSizeMB

I'd like to produce a dashboard if one doesnt already exist to answer the following questions:

  1. What is the oldest data in each index? eg its 183 days old
  2. How much of the total allotted space is each index using for its hot and cold stores.: eg 98% of Hot and 15% of Cold is being used for this index.
  3. How much of the physical disks have we allocated to indexes. For example if all indexes were full, have we allocated 150% of the physical space available? All indexes sit on a HOT disk and a COLD disk. Eg 98% of hot disk is allocated, 150% of cold disk space is allocated
  4. At the current rate of ingestion, what would the retention be if we used 100% of all allocated space available for the index. eg 360 days
  5. What is limiting our retention - is it our maxTotalDataSizeMB or frozenTimePeriodInSecs for each index.

Thanks for your help.

Here is a screenshot showing a typical index definition that is pushed out to our index cluster.

alt text

0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi davidwaugh,
I think that you should see the Monitoring Console App and check if it solves all your needs.
In addition there's the Index Usage App ( https://splunkbase.splunk.com/app/4086/ ) that could be very useful for your needs.
If there are some need that you cannot solve with them, let me know and surely you'll have the support you need.

Bye.
Giuseppe

View solution in original post

roseg001
New Member

please can some one help me splunk retention policy stanza for 80 days

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi davidwaugh,
I think that you should see the Monitoring Console App and check if it solves all your needs.
In addition there's the Index Usage App ( https://splunkbase.splunk.com/app/4086/ ) that could be very useful for your needs.
If there are some need that you cannot solve with them, let me know and surely you'll have the support you need.

Bye.
Giuseppe

davidwaugh
Path Finder

Thanks very much. I've just installed Index Usage and have used the Monitoring Console. I think it will take a few days to ingest the data for the dashbaords so will let you know.
Thanks for your help.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi davidwaugh,
if you're (o when you'll be) satisfied by this answer, please accept and/or upvote it.
Bye.
Giuseppe

0 Karma

davidwaugh
Path Finder

Thanks Index Usage was the answer. Great app!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...