Reporting

How do I stop old records being deleted?

simonsjw
Engager

I have a set of log records dating from 2009 to 2011. I upload them to Splunk and set MAX_DAYS_AGO=10,000 as well as setting the Tsidx Retention Policy to disabled.

However, whenever I restart splunk, the records are deleted.

Does anyone know how to stop the old records being deleted?

Thanks and regards,

Simon

0 Karma

gjanders
SplunkTrust
SplunkTrust

There are a couple of settings to check here, as per the indexes.conf documentation , you mention 2009 to 2011, the default frozenTimePeriodInSecs is 6 years, so you can either change that for every index under:

[default]

Or per-index.

The other setting to check is your maxTotalDataSizeMB which is set to 500,000MB by default, if that is exceeded you will also have data frozen.

To check what the current sizing is of the indexes refer to the monitoring console of Splunk , it has a panel on index information which will show the frozenTimePeriodInSecs and current oldest data et cetera.

If the data has been frozen already the only restoration method is restore from backup as per the documentation

A search such as (if you have a Linux based indexer):

index=_internal sourcetype=splunkd source="/opt/splunk/var/log/splunk/splunkd.log" "BucketMover - will attempt to freeze"

Will show all bucket freezing activity on the indexers...I have an example query here to detect when data is frozen due to reaching the index size limits but not the frozenTimePeriodInSecs.
To make the above search more efficient you may wish to add a host= clause for your indexer names...

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

MAX_DAYS_AGO is a setting that only affects processing at indexing time. It is not a setting that controls data retention. Refer to the doc link posted below for configuring index retention policies.

0 Karma

schollaert
Explorer

Have a look at the way buckets (records) are rolled to frozen (deleted) https://docs.splunk.com/Documentation/Splunk/7.0.0/Indexer/Setaretirementandarchivingpolicy

Best regards,

Jan

simonsjw
Engager

Thanks so much Jan - I read the docs and can see that I needed to update frozenTimePeriodInSecs in my indexes.conf file.

However, it appears there are many and although I believe I've updated the relevant one, my records are still being deleted.

Appreciate you taking the time - definitely the right track.

Simon

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...