I have a set of log records dating from 2009 to 2011. I upload them to Splunk and set MAX_DAYS_AGO=10,000 as well as setting the Tsidx Retention Policy to disabled.
However, whenever I restart splunk, the records are deleted.
Does anyone know how to stop the old records being deleted?
Thanks and regards,
There are a couple of settings to check here, as per the indexes.conf documentation , you mention 2009 to 2011, the default frozenTimePeriodInSecs is 6 years, so you can either change that for every index under:
The other setting to check is your maxTotalDataSizeMB which is set to 500,000MB by default, if that is exceeded you will also have data frozen.
To check what the current sizing is of the indexes refer to the monitoring console of Splunk , it has a panel on index information which will show the frozenTimePeriodInSecs and current oldest data et cetera.
If the data has been frozen already the only restoration method is restore from backup as per the documentation
A search such as (if you have a Linux based indexer):
index=_internal sourcetype=splunkd source="/opt/splunk/var/log/splunk/splunkd.log" "BucketMover - will attempt to freeze"
Will show all bucket freezing activity on the indexers...I have an example query here to detect when data is frozen due to reaching the index size limits but not the frozenTimePeriodInSecs.
To make the above search more efficient you may wish to add a host= clause for your indexer names...
MAX_DAYS_AGO is a setting that only affects processing at indexing time. It is not a setting that controls data retention. Refer to the doc link posted below for configuring index retention policies.
Thanks so much Jan - I read the docs and can see that I needed to update frozenTimePeriodInSecs in my indexes.conf file.
However, it appears there are many and although I believe I've updated the relevant one, my records are still being deleted.
Appreciate you taking the time - definitely the right track.