Getting Data In

Delay in Splunk purging old events

danielwan
Explorer

My Splunk is a single Splunk 6.5.x instance, which needs to retain the last 30 days events, so I configured frozenTimePeriodInSecs = 2592000 in indexes.conf. But it does not work fine all the time.

What I could tell is my indexes keep growing, and search with "latest=-30d" shows up some events sometimes. When the index size reaches the maximum index size which was configured in the index creation, or when I restart Splunk instance, the index size decreases to nearly half of the max index size.

Is there any idea of why there is so significant delay for Splunk purging old events? and how to fix it?

0 Karma
1 Solution

harsmarvania57
SplunkTrust
SplunkTrust

Hi @danielwan,

Based on documentation of indexes.conf, index will remove data from index based on 2 parameters frozenTimePeriodInSecs OR maxTotalDataSizeMB whichever hit first.

Now splunk stores data in hot, warm and cold buckets. In your case when you set frozenTimePeriodInSecs to 2592000 it will remove those warm or cold bucket which will have all events older than frozenTimePeriodInSecs .

IMPORTANT: Every event in the DB must be older than frozenTimePeriodInSecs
  before it will roll.

So let's day one of the bucket contains earliest event 45 days older and latest event is 25 days older then this bucket(DB) will not remove and when you will search you will able to search data older than 30 days from that bucket, this bucket will remove when all events in that bucket are older than frozenTimePeriodInSecs

Now when you restart splunk it will roll hot bucket to warm and warm to cold based on your indexes.conf configuration and in this case if any hot bucket contain events older than 30 days then it will roll hot bucket to warm and then immedeiatly remove that bucket and due to that your index size decrease suddenly.

Thanks,
Harshil

View solution in original post

gcusello
Legend

Hi danielwan,
event deletion is managed at bucket level, so when the latest event of a bucket is out of retention period bucket will be frozen or deleted.
This means that you can have online some events older that the retention period.
Bye.
Giuseppe

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Hi @danielwan,

Based on documentation of indexes.conf, index will remove data from index based on 2 parameters frozenTimePeriodInSecs OR maxTotalDataSizeMB whichever hit first.

Now splunk stores data in hot, warm and cold buckets. In your case when you set frozenTimePeriodInSecs to 2592000 it will remove those warm or cold bucket which will have all events older than frozenTimePeriodInSecs .

IMPORTANT: Every event in the DB must be older than frozenTimePeriodInSecs
  before it will roll.

So let's day one of the bucket contains earliest event 45 days older and latest event is 25 days older then this bucket(DB) will not remove and when you will search you will able to search data older than 30 days from that bucket, this bucket will remove when all events in that bucket are older than frozenTimePeriodInSecs

Now when you restart splunk it will roll hot bucket to warm and warm to cold based on your indexes.conf configuration and in this case if any hot bucket contain events older than 30 days then it will roll hot bucket to warm and then immedeiatly remove that bucket and due to that your index size decrease suddenly.

Thanks,
Harshil

Get Updates on the Splunk Community!

Testing out the OpenTelemetry Collector With raw Data

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

New Cloud Intrusion Detection System Add-on for Splunk

In July 2022 Splunk released the Cloud IDS add-on which expanded Splunk capabilities in security and data ...

Happy CX Day to our Community Superheroes!

Happy 10th Birthday CX Day!What is CX Day? It’s a global celebration recognizing innovation and success in the ...