Reporting

How do I stop old records being deleted?

simonsjw
Engager

I have a set of log records dating from 2009 to 2011. I upload them to Splunk and set MAX_DAYS_AGO=10,000 as well as setting the Tsidx Retention Policy to disabled.

However, whenever I restart splunk, the records are deleted.

Does anyone know how to stop the old records being deleted?

Thanks and regards,

Simon

0 Karma

gjanders
SplunkTrust
SplunkTrust

There are a couple of settings to check here, as per the indexes.conf documentation , you mention 2009 to 2011, the default frozenTimePeriodInSecs is 6 years, so you can either change that for every index under:

[default]

Or per-index.

The other setting to check is your maxTotalDataSizeMB which is set to 500,000MB by default, if that is exceeded you will also have data frozen.

To check what the current sizing is of the indexes refer to the monitoring console of Splunk , it has a panel on index information which will show the frozenTimePeriodInSecs and current oldest data et cetera.

If the data has been frozen already the only restoration method is restore from backup as per the documentation

A search such as (if you have a Linux based indexer):

index=_internal sourcetype=splunkd source="/opt/splunk/var/log/splunk/splunkd.log" "BucketMover - will attempt to freeze"

Will show all bucket freezing activity on the indexers...I have an example query here to detect when data is frozen due to reaching the index size limits but not the frozenTimePeriodInSecs.
To make the above search more efficient you may wish to add a host= clause for your indexer names...

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

MAX_DAYS_AGO is a setting that only affects processing at indexing time. It is not a setting that controls data retention. Refer to the doc link posted below for configuring index retention policies.

0 Karma

schollaert
Explorer

Have a look at the way buckets (records) are rolled to frozen (deleted) https://docs.splunk.com/Documentation/Splunk/7.0.0/Indexer/Setaretirementandarchivingpolicy

Best regards,

Jan

simonsjw
Engager

Thanks so much Jan - I read the docs and can see that I needed to update frozenTimePeriodInSecs in my indexes.conf file.

However, it appears there are many and although I believe I've updated the relevant one, my records are still being deleted.

Appreciate you taking the time - definitely the right track.

Simon

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...