Splunk data deletion from retention policy (frozenTimePeriodInSecs)

[uagraw01]

Hello Splunker!!

Here’s your question rewritten in a business context and structured in points:

1. Objective: To free up disk space by deleting 1 month of data from a specific Splunk index containing 1 year of data.

2. Key Considerations:
- How can we verify that the deletion of 1 month of data from Splunk indexes is successful?
- How long does Splunk typically take to delete this amount of data from the indexes?
- Is there a way to monitor or observe the deletion of old buckets or data using the Splunk UI (via SPL queries)?

Thanks in advance!!

[jawahir007]

Hi @uagraw01 : The Splunk SPL command below might be helpful for you...

|dbinspect index=your_index

In Splunk, the dbinspect command is used to gather detailed metadata about the index buckets in a specified index or set of indexes. This command provides information about the state, size, and other characteristics of the index buckets, which can help with monitoring storage, troubleshooting indexing issues, and understanding how Splunk is managing the data on disk.

Key Information Provided by dbinspect:

• Bucket ID: A unique identifier for each bucket.
• Index Name: The index to which the bucket belongs.
• Start and End Time: The time range of events contained within the bucket.
• Bucket State: The current state of the bucket, such as:
  • hot: Currently being written to.
  • warm: Closed and searchable.
  • cold: Moved to colder storage.
  • frozen: No longer searchable, either deleted or archived.
  • thawed: Restored from archive (frozen) and searchable again.
• Size on Disk: The storage size of the bucket.
• Event Count: The number of events contained in the bucket.
• Size before Compression: The size of the bucket before compression.

Example Use Cases:

1. Search for Buckets by State: To filter for buckets in a specific state (e.g., cold or warm buckets), you can modify the query like this:

| dbinspect index=your_index
| search state="warm" OR state="cold"
| table bucketId, index, startEpoch, endEpoch, state, sizeOnDiskMB

This query filters for buckets that are either in the warm or cold state and displays useful details such as the bucket ID, size, and time range

2. Analyze Bucket Sizes: You can use dbinspect to analyze how much storage each bucket is consuming and understand your disk usage:

| dbinspect index=your_index
| stats sum(sizeOnDiskMB) as totalSize by index

This query calculates the total disk size used by the specified index.

3. Find Old Buckets: To find the oldest buckets in an index based on their time ranges:

| dbinspect index=your_index
| sort startEpoch
| table bucketId, index, startEpoch, endEpoch, state, sizeOnDiskMB

This helps to identify which buckets contain the oldest data and may be candidates for deletion based on your data retention policies.

------

If you find this solution helpful, please consider accepting it and awarding karma points !!

[uagraw01]

@jawahir007 its very tedious to specify the status of 11 month old bucket start and end time. 

suppose data age of event is 427 days and I want to delete 30 days of data then it data age now would be 397.Now how can I identify the epoch of start and end time of bucket ?

[jawahir007]

The following SPL will also provide the bucket age in days. Once you have this information, you can configure the frozenTimePeriodInSecs attribute to specify how long to retain data and when to move it out. Run the query again to verify if the data has been removed

|dbinspect index=your_index|eval end_days=round((now()-endEpoch)/86400,0)
|eval start_days=round((now()-startEpoch)/86400,0)|table start_days,end_days,*

[uagraw01]

@jawahir007 Thanks for providing this SPL. So My retention is currently at 437 days and I have to reduce the retention by 30 days, which means I have to set the retention at 407 days. I also need to adjust this in above query earliest=-437d latest=-407d. Is that right ?

[jawahir007]

• Run the query for All Time to identify the oldest bucket in the specified index. (Just to get information)

• The fields start_days and end_days represent the time range of events contained within each bucket.

• Sort the buckets by end_days in descending order to find the oldest bucket in that index.

• For example, if the end_days value is 500 and you only want to retain 400 days of data, configure the following parameter in your index settings

frozenTimePeriodInSecs = 34560000      #(Seconds equivalent to 400days)

------

If you find this solution helpful, please consider accepting it and awarding karma points !!

[isoutamo]

Here is some old posts which explain why this works like it works.

• https://community.splunk.com/t5/Deployment-Architecture/Right-number-and-size-of-hot-warm-cold-bucke...
• https://community.splunk.com/t5/Getting-Data-In/Indexes-configuration/m-p/564271
• https://community.splunk.com/t5/Splunk-Enterprise/Splunk-shows-only-9-months-270-days-data-How-do-I-...
• https://community.splunk.com/t5/Deployment-Architecture/Hot-Warm-Cold-bucket-sizing-How-do-I-set-up-...
• https://community.splunk.com/t5/Splunk-Enterprise/Why-is-the-age-of-the-data-larger-than-the/m-p/548...
• https://community.splunk.com/t5/Deployment-Architecture/How-do-I-expire-a-bucket-with-future-events/...
• https://community.splunk.com/t5/Deployment-Architecture/script-to-convert-the-bucket-name-to-the-tim...

And there are lot more with searching that issue from Google.

[uagraw01]

@isoutamo Thankd for sharing me all the valuable accepted solution link. I will go through each of the link.

[isoutamo · Accepted solution]

Hi

here is how this should work, but as there are some "magic" how those events are in buckets it's not as simple and doable as you could expect 😞

2.1) Just query from that index if there is events before that retention time. But quite probably there are some events still. The reason for that is that smallest storage artifact/objet is bucket not an individual event. And one bucket can contain events from very large time span.

2.2) This is totally dependent of amount of data, your instances sizes and other resource aspects which always depends.

2.3) You could look from MC (monitoring console) Settings -> MC -> Indexing. -> Indexes and Volumes -> Index Detail: Deployment. This dashboard shows that information.

r. Ismo

[uagraw01]

@isoutamo  Yes From MC i can take refrence of number of bucket counts atleast.

[isoutamo]

Here 

is what I can see on my own test instance. As you see there are a lot more information than only count of buckets.

You can also click that magnifying glass and then you see the exact SPL query how to get this information. Then you can modify this for better answer your needs.