topic Re: Process for Data retenetion policies in Getting Data In

Process for Data retenetion policies

splunkgk — Tue, 20 Jun 2017 05:03:58 GMT

Hi,

I wanted to apply data retention policy on splunk enterprise for the first time (as of now this is default) as per below criteria.

All indexes will have the last 12 months of data available for search "hot bucket"
After this, data will roll to the "warm bucket" for 3 months.
Then the "cold bucket" for 3 months.
After 18 months all incoming data is effectively irrecoverable from Splunk.

This will be great of someone cloud share me how exactly this can be done and how to set the indexes.conf.

-Thanks

Re: Process for Data retenetion policies

gcusello — Tue, 20 Jun 2017 11:52:47 GMT

Hi splunkgk,
roll from hot to warm bucket doesn't move logs so it isn't a good idea to have large hot buckets (see http://docs.splunk.com/Documentation/Splunk/6.6.1/Indexer/HowSplunkstoresindexes ).
Anyway, parameters to roll buckets are described in https://docs.splunk.com/Documentation/Splunk/6.6.1/Admin/Indexesconf

Bye.
Giuseppe

Re: Process for Data retenetion policies

splunkgk — Wed, 21 Jun 2017 09:13:02 GMT

HI Giuseppe, Thanks for reply. I have gone though the settings and a question here, I have around 8 indexes which are configured in my "search" app located under /SPLUNLK_HOME/etc/apps/search/local/indexes.conf. My question here is how do i set rolling parameter which should effect on all indexes.

Or is that i need to add a parameter in indivisual [index] blocks in each apps?

I want to apply the above criteria (Base question) on indexes in all my apps globally. Could you please suggests me where can i teak the settings?

-thanks

Re: Process for Data retenetion policies

gcusello — Wed, 21 Jun 2017 09:47:02 GMT

Hi splunkgk,
I usually put my indexes.conf in a dedicated App, but it depends by architecture: e.g. a clustered architecture requires that indexes are managed by Master Node and I prefer to have only one indexes.conf, instead, if you have more than one Indexer, you could manage them deploying a TA (containing indexes.conf) by Deployment Server; if instead you have a Stand Alone server, maybe it's simpler to insert indexes.conf in each App, it's your own approach.

Anyway, these Parameters must be modified in each stanza of indexes.conf files.

Bye.
Giuseppe

Re: Process for Data retenetion policies

splunkgk — Tue, 29 Sep 2020 14:37:43 GMT

Hi Giuseppe, Thank your your reply.
My current architecture is stand alone server and i do not have a clustered index. indexes.conf are placed in individual app folder under /SPLUNK_HOME/etc/apps/search/local/indexes.conf.

Here is below my one of index which is a having large data under "search" app.

[index_name]
coldPath = $SPLUNK_DB/index_name/colddb
homePath = $SPLUNK_DB/index_name/db
thawedPath = $SPLUNK_DB/index_name/thaweddb
maxTotalDataSizeMB = 750000

so I would like to apply data retention on this index as mentioned in the question as (roll must happen as HOT Bucket =12 months, WARM bucket= 3 months and COLD bucket= 3 months, over the period of 18 months data must not reside on splunk volume)

is the below setting need to set on each of indexes.conf on each apps?
maxHotIdleSecs = 31536000( as i wanted to retain 12 months of data in HOT bucket)
maxWarmDBCount = 300
frozenTimePeriodInSecs = 7884000 (90 days in sec, cold to frozen)
coldToFrozenDir = /archive/myindex ( after 90 days, index goes here)

Thanks,

Re: Process for Data retenetion policies

splunkgk — Tue, 29 Sep 2020 14:49:47 GMT

Hi Giuseppe,

is the below setting in /SPLUNK_HOME/etc/system/local/indexes.conf will works on all of my current indexes which are stored in /SPLUNK_HOME/var/lib/splunk?

[default]
frozenTimePeriodInSecs = 31536000 [1 year]
enableTsidxReduction = true
timePeriodInSecBeforeTsidxReduction = 345600 [4 days]

Or should i copy this stanza in individual indexes ?

One thing I should take care that, The data must be stay in cold bucket for 3 months before this rolled to FREEZE.

How do make sure the above retention policy works with my criteria

-Thanks

Re: Process for Data retenetion policies

gcusello — Tue, 11 Jul 2017 11:01:26 GMT

Hi splunkgk,
you have to put your options in every indexes.conf stanzas.

I don't like to have a so large hot area because hot buckets are written and read at the same time, instead warm buckets are only read and this is more efficient in searches (see http://docs.splunk.com/Documentation/Splunk/6.6.2/Indexer/HowSplunkstoresindexes ).

the cold period depends from the global retention defined in frozenTimePeriodInSecs and from how long that buckets are hot and warm.

Bye.
Giuseppe