Getting Data In

Retention policy for 30 days (15 searchable + 15 frozen)

dstoev
Explorer

Hello all,

I'm trying to setup the following retention policy:

15 days of events to be searchable (hot/warm/cold - it doesn't matter) + 15 days of data to be frozen (archived). So always I would like to have 30 days of data (of course 15 days should be thaw out first if I want to use them).

So for example:

Events/data from 01.01.2020 till 15.01.2020 to be frozen (archived), the data from 16.01.2020 till 31.01.2020 to be searchable. And of course this will shift day by day.

I've read a lot for frozenTimePeriodInSecs, maxHotSpanSecs, maxHotIdleSecs and etc. but I'm very confused if my scenario can be achieved with these parameters.

Please note that I'm trying to accomplish this scenario WITHOUT any dependency if the size for buckets, indexes and etc.

My architecture is:
10-15 windows hosts sending some logs to 1 Splunk indexer (7.1.4).

Thank you in advance!

0 Karma
1 Solution

gcusello
Legend

Hi @dstoev,
with frozenTimePeriodInSecs you define how long logs are in your Splunk indexes, in your example the first 15 days.
After this period you can discard these logs or put them in frozen state using a script that copies frozenable buckets in another folder and then discard them from on line buckets, but after this operation you cannot manage in Splunk the copied frozen buckets, in other words you cannot define retention for frozen logs in Splunk.
You could try to schedule a script on the operative system that deletes folders older that 30 days but only outside Splunk.

Here you can find detailed information about Splunk retention policies https://docs.splunk.com/Documentation/Splunk/8.0.1/Indexer/Setaretirementandarchivingpolicy .

Ciao.
Giuseppe

View solution in original post

nickhills
Ultra Champion

If AWS is an available option, you could consider using "Dynamic Data Self Storage" (see indexes.conf)

You set your frozen period as required (15 days) and Splunk will freeze the data and move it to a bucket in S3.
Configure a bucket policy to remove data (or maybe archive it to Glacier if you desire) 15 days after its written to s3

The process is then: Hot->Warm->Cold (for 15 days) -> Archive to S3 ->After 15 Days >[Delete|Archive to Glacier]

If my comment helps, please give it a thumbs up!
0 Karma

dstoev
Explorer

Yep, as @gcusello said - this can be only achieved using ...3rd party solutions.

Thank you!

0 Karma

gcusello
Legend

Hi @dstoev,
with frozenTimePeriodInSecs you define how long logs are in your Splunk indexes, in your example the first 15 days.
After this period you can discard these logs or put them in frozen state using a script that copies frozenable buckets in another folder and then discard them from on line buckets, but after this operation you cannot manage in Splunk the copied frozen buckets, in other words you cannot define retention for frozen logs in Splunk.
You could try to schedule a script on the operative system that deletes folders older that 30 days but only outside Splunk.

Here you can find detailed information about Splunk retention policies https://docs.splunk.com/Documentation/Splunk/8.0.1/Indexer/Setaretirementandarchivingpolicy .

Ciao.
Giuseppe

dstoev
Explorer

Hi @gcusello, thank you for this. May I ask if I'm right about the following, considering the following indexes.conf file:

[my_index]
maxHotSpanSecs = 1296000
frozenTimePeriodInSecs = 1296000
coldToFrozenDir = /some/dir

This will mean that no matter what, the hot bucket will roll out to warm after 15 days (NOT 15 days of events but simply 15 calendar days), and buckets won't roll to forzen until the newest event is older than frozenTimePeriodInSecs ? Note - I do not have maxDataSize set.

Thank you!

0 Karma

gcusello
Legend

Hi @dstoev,
it's correct not setting maxDataSize because you couldn't respect time retention policy.
maxHotSpanSecs isn't relevant for retention because is related to Hot state, so if you want, you can set it, I usually leave the default values.
coldToFrozenDir is what you need to put offline frozen buckets, but (as I said) you have to manage retention of frozen buckets outside Splunk.

Ciao.
Giuseppe

0 Karma

dstoev
Explorer

Hi @gcusello ,

Thank you!

0 Karma

jethrop
Explorer

Hey, definitely achievable. have you tried using this site: https://splunk-sizing.appspot.com/

0 Karma

dstoev
Explorer

Actually I have, but still I'm not sure how to do it. I'm on the same opinion as @gcusello (that this cannot be achieved except using some script), but I've decided to ask here just to be sure that I'm not missing anything.

0 Karma
Get Updates on the Splunk Community!

Happy CX Day to our Community Superheroes!

Happy 10th Birthday CX Day!What is CX Day? It’s a global celebration recognizing innovation and success in the ...

Check out This Month’s Brand new Splunk Lantern Articles

Splunk Lantern is a customer success center providing advice from Splunk experts on valuable data insights, ...

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...