topic Re: Is it possible for data to be searchable indefinitely? in Getting Data In

Is it possible for data to be searchable indefinitely?

takashi6 — Mon, 15 Jun 2020 06:27:05 GMT

Hello Experts,

I understand we can use "frozenTimePeriodInSecs" to move the data to a frozen state and the data becomes unsearchable once that happens.

We have a requirement that the data remains searchable indefinitely. The moving data to a unsearchable state takes place on a regular basis but only after receiving an approval from a set of people and we can't set a certain retention period to our indexes.

Is it possible we set someone like "indefinite" or "infinite" to the frozenTimePeriodInSecs?

Re: Is it possible for data to be searchable indefinitely?

gcusello — Mon, 15 Jun 2020 06:39:48 GMT

Hi @takashi6 ,

as you can see at https://docs.splunk.com/Documentation/Splunk/8.0.4/Admin/Indexesconf

frozenTimePeriodInSecs = <nonnegative integer>
* The number of seconds after which indexed data rolls to frozen.
* If you do not specify a 'coldToFrozenScript', data is deleted when rolled to
  frozen.
* NOTE: Every event in a bucket must be older than 'frozenTimePeriodInSecs'
  seconds before the bucket rolls to frozen.
* The highest legal value is 4294967295.
* Default: 188697600 (6 years)

The default value is 188697600 (6 years), the highest legal value is 4294967295, that means around 136 years, is it sufficient to answer to your requirement?

In other words, there isn't an "indefinite" value, but you can use an high value that gives you the same result.

In addition, if you want you can also frozen the deleted values using a script at the end of the retentio0n period to store frozen data out of on line data, but they are still searcheable.

Ciao.

Giuseppe

Re: Is it possible for data to be searchable indefinitely?

takashi6 — Mon, 15 Jun 2020 08:10:16 GMT

Thank you @gcusello for your valuable input.

I understand I need to put a nonnegative integer AND I can input a really, really high value.

May I ask - what would happen if I don't include "frozenTimePeriodInSecs" in the .conf for a particular Index?

What retention period will be in effect for the index?

Re: Is it possible for data to be searchable indefinitely?

gcusello — Mon, 15 Jun 2020 11:02:14 GMT

Hi @takashi6

without frozenTimePeriodInSecs, you have the default value

Default: 188697600 (6 years)

Only one little hint: having on line all the data is expensive in terms of storage, backup and response time: so, analyze the possibility to maintain on line a subset of data (e.g. one year) and put the other data in frozen state that you can search non immediately but in a quick time, or put the oldest data in a less performant storage.

Ciao.

Giuseppe

Re: Is it possible for data to be searchable indefinitely?

takashi6 — Mon, 15 Jun 2020 13:39:37 GMT

Thanks again for your response and insight, @gcusello

I've understood the behavior and suggestions. now I'm closing this questions - as answered!