topic Re: Data retention in Splunk Search

Data retention

dani9 — Thu, 14 Nov 2019 13:57:31 GMT

Where must the data retention be settled in indexer or in my case distributed environment in search head?
Then seen that it must be setted in file indexes.conf but it S present just in etc/system/default but we know we don't have to edit files in default folder how can I do that? Do I create a file in local and after splunk will think to update the default folder?

Re: Data retention

gcusello — Wed, 30 Sep 2020 02:59:06 GMT

Hi @dani9,
to set a data retention for an index, you have to work only on Indexers, or (if you have an Indexers cluster) on Master Node, never Search Heads.
You have to create an indexes.conf file in $SPLUNK_HOME/system/local (never default!), or better, create a dedicated Add-On (called e.g. TA_indexers) in with there's this file to insert in $SPLUNK_HOME/etc/apps.

To set data retention you have to insert in indexes.conf:

[index_name]
frozenTimePeriodInSecs = integer

Remember that if you add or modify a conbf file you have to restart Splunk.

frozenTimePeriodInSecs = <nonnegative_integer>
* The number of seconds after which indexed data rolls to frozen.
* If you do not specify a 'coldToFrozenScript', data is deleted when rolled to   frozen.
* NOTE: Every event in a bucket must be older than 'frozenTimePeriodInSecs' seconds before the bucket rolls to frozen.
* The highest legal value is 4294967295.
* Default: 188697600 (6 years)

for more info see at:
https://docs.splunk.com/Documentation/Splunk/8.0.0/Admin/Indexesconf
https://docs.splunk.com/Documentation/Splunk/8.0.0/Indexer/Setaretirementandarchivingpolicy

Ciao.
Giuseppe

Re: Data retention

dani9 — Thu, 14 Nov 2019 14:59:05 GMT

How often maxtotaldatasize is set?

Re: Data retention

dani9 — Thu, 14 Nov 2019 15:04:01 GMT

NOTE: Every event in a bucket must be older than 'frozenTimePeriodInSecs' seconds before the bucket rolls to frozen

This means that i have to set each bucket like telemetry, main, summary etc.. Older that frozen field?

Re: Data retention

gcusello — Thu, 14 Nov 2019 15:15:11 GMT

Hi @dani9,
I don't like to use the dimension of the index because I usually have compliance need and I prefer to use frozenTimePeriodInSecs.

Anyway, if you prefer to use maxtotaldatasize, you can use it as I described in my answer.

Ciao.
Giuseppe

Re: Data retention

dani9 — Thu, 14 Nov 2019 15:18:43 GMT

The fact is just insert the field frozen the retention works? Because in the guides advices to set also maxtotalsize and coldfrozentodir, without these it works the same?
How can I see if data retention is effectively working?

Re: Data retention

gcusello — Thu, 14 Nov 2019 16:17:52 GMT

Hi @dani9,
a bucket is rolled to frozen (in other words deleted if you haven't any script) when the latest event exceed the retention period.
You have to set the retention period for each index or you can set a default value, I don't like this because I like to have a full control on data retention!
Anyway, it's important to intervene on the largest indexes: e.g. _internal is a large index and it's udeful to set retention, _audit it isn't.

Ciao.
Giuseppe