topic Re: Do I need frozen storage? in Getting Data In

Do I need frozen storage?

rewritex — Fri, 30 Jun 2017 17:20:05 GMT

My retention policy has drastically changed and we are utilizing syslog as a main retention source.
On the Splunk side, I don't have a need for frozen data so I am planning just to zero it out

frozenTimePeriodInSecs = 0

Q: Is this correct?
Q: Would I just put the above line in each index.conf stanza or is there a global spot?

My goal is to just have hot/warm searchable and no cold/frozen. Future plans may be to use Splunk as a retention source and offload frozen data to another volume/location.

Thank You,
Sean

Re: Do I need frozen storage?

jkat54 — Fri, 30 Jun 2017 18:14:18 GMT

When data roles to frozen it is deleted by default.

So setting that to 0 would just roll everything to frozen immediately I suppose. Typically 0 = indefinitively but it's not documented in indexes.conf spec as such.

It's really tricky to "disable" cold and I recommend you just keep it. If you must remove it, the easiest method is probably going to be a warmToColdScript that just deletes the data.

Still it would probably be better to set frozenTimePeriodInSecs to whatever value you want to keep hot/warm.., say 90 days but in seconds (90*86400) and then let Splunk do its automagic thing in relation to buckets. Also keep the path to the cold storage on the same hot/warm disk and there shouldn't be any performance impact.

Re: Do I need frozen storage?

richgalloway — Fri, 30 Jun 2017 18:21:52 GMT

Presumably, you have coldToFrozenScript or coldToFrozenDir defined to freeze your data. If you remove those attributes, data will be deleted when it ages out of cold.
To not retain cold data, define a warmToColdScript for each index that simply deletes the buckets.
Note that you must restart Splunk for these changes to take effect.