Splunk Search

Data retention

dani9
Explorer

Where must the data retention be settled in indexer or in my case distributed environment in search head?
Then seen that it must be setted in file indexes.conf but it S present just in etc/system/default but we know we don't have to edit files in default folder how can I do that? Do I create a file in local and after splunk will think to update the default folder?

Tags (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @dani9,
to set a data retention for an index, you have to work only on Indexers, or (if you have an Indexers cluster) on Master Node, never Search Heads.
You have to create an indexes.conf file in $SPLUNK_HOME/system/local (never default!), or better, create a dedicated Add-On (called e.g. TA_indexers) in with there's this file to insert in $SPLUNK_HOME/etc/apps.

To set data retention you have to insert in indexes.conf:

[index_name]
frozenTimePeriodInSecs = integer

Remember that if you add or modify a conbf file you have to restart Splunk.

frozenTimePeriodInSecs = <nonnegative_integer>
* The number of seconds after which indexed data rolls to frozen.
* If you do not specify a 'coldToFrozenScript', data is deleted when rolled to   frozen.
* NOTE: Every event in a bucket must be older than 'frozenTimePeriodInSecs' seconds before the bucket rolls to frozen.
* The highest legal value is 4294967295.
* Default: 188697600 (6 years)

for more info see at:
https://docs.splunk.com/Documentation/Splunk/8.0.0/Admin/Indexesconf
https://docs.splunk.com/Documentation/Splunk/8.0.0/Indexer/Setaretirementandarchivingpolicy

Ciao.
Giuseppe

0 Karma

dani9
Explorer

NOTE: Every event in a bucket must be older than 'frozenTimePeriodInSecs' seconds before the bucket rolls to frozen

This means that i have to set each bucket like telemetry, main, summary etc.. Older that frozen field?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @dani9,
a bucket is rolled to frozen (in other words deleted if you haven't any script) when the latest event exceed the retention period.
You have to set the retention period for each index or you can set a default value, I don't like this because I like to have a full control on data retention!
Anyway, it's important to intervene on the largest indexes: e.g. _internal is a large index and it's udeful to set retention, _audit it isn't.

Ciao.
Giuseppe

0 Karma

dani9
Explorer

How often maxtotaldatasize is set?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @dani9,
I don't like to use the dimension of the index because I usually have compliance need and I prefer to use frozenTimePeriodInSecs.

Anyway, if you prefer to use maxtotaldatasize, you can use it as I described in my answer.

Ciao.
Giuseppe

0 Karma

dani9
Explorer

The fact is just insert the field frozen the retention works? Because in the guides advices to set also maxtotalsize and coldfrozentodir, without these it works the same?
How can I see if data retention is effectively working?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...