- Splunk Answers
- :
- Splunk Platform Products
- :
- Splunk Enterprise
- :
- How to set a retention policy to delete any data i...
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
How to set a retention policy to delete any data in a certain index after 30 days?
Simply put, I'd like to delete any data in a certain index after 30 days. I set the retention policy to 30 days, but I still have events from over 30 days ago.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Following are the configurations that you can set to keep data only for exactly 30 days. The data in Splunk is written in hot buckets. The name of the bucket includes the min and max of the _time of the data included in it. When Splunk has to apply data retention, based on age, then it picks the buckets whose max _time value is older than retention period. So if a buckets contains data for 3 day period, it won't get deleted until day 33 and you'd see some partial data for day 31/32/33 (that's what is happening in your case).
The below setting will enforce Splunk to contain maximum of 1 day worth of data in the buckets, so that buckets can retire on exact day basis. NOTE: If you set this too small, you can get an explosion of hot/warm buckets in the filesystem. Default to 90 days.
Indexes.conf on your Indexers
[yourindex]
...other settings...
maxHotSpanSecs = 86400 # 1 day
frozenTimePeriodInSecs = 2592000 # 30 days
See more info of retention policies here:
http://docs.splunk.com/Documentation/Splunk/6.4.1/Indexer/Setaretirementandarchivingpolicy
https://wiki.splunk.com/Deploy:BucketRotationAndRetention
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
So if I create an indexes.conf in '.\Splunk\etc\system\local' and change it to this (and restart splunk service), it'll work?
[wineventlog]
coldPath = $SPLUNK_DB\wineventlog\colddb
homePath = $SPLUNK_DB\wineventlog\db
thawedPath = $SPLUNK_DB\wineventlog\thaweddb
maxHotSpanSecs = 2592000
frozenTimePeriodInSecs = 2592000
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
It should.. You've stand alone Indexer? Do you have one indexer OR many? The changes from location 'Splunk\etc\system\local' works but the preferred location is etc/apps under some custom app (for better portability).
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Oops, I'm now noticing it dropped from 4 years to 2 years. Perhaps it's working...
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The maxHotSpanSecs value you set is 30 days (I suggested 1 day) ,as well as frozenTimePeriodInSecs. So total data will be retained only for 30 days. That is what you intend to do.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Have you set frozenTimePeriodInSecs to 30 days in seconds?
More about it at How is frozenTimePeriodInSecs applied?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I don't have a frozen path specified.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
That's fine as by default, when reaching the frozen state, the data is being deleted.
.conf24 | Registration Open!
ICYMI - Check out the latest releases of Splunk Edge Processor
Introducing the 2024 SplunkTrust!
