How does Splunk handle the frozen storage ?

[hazem]

Hello,
 My index configuration is provided below, but I have a question regarding frozenTimePeriodInSecs = 7776000. I have configured Splunk to move data to frozen storage after 7,776,000 seconds (3 months). Once data reaches the frozen state, how can I control the frozen storage if the frozen disk becomes full? How does Splunk handle the frozen storage in such scenarios?

[custom_index]
repFactor = auto
homePath = volume:hot/$_index_name/db
coldPath = volume:cold/$_index_name/colddb
thawedPath = /opt/thawed/$_index_name/thaweddb
homePath.maxDataSizeMB = 1664000
coldPath.maxDataSizeMB = 1664000
maxWarmDBCount = 200
frozenTimePeriodInSecs = 7776000
maxDataSize = auto_high_volume
coldToFrozenDir = /opt/frozen/custom_index/frozendb

[PickleRick · Accepted solution]

Splunk does _not_ handle frozen storage. It's up to you.

As soon as splunkd pushes a bucket out to frozen it loses all interest in further well-being of that bucket and/or storage it's on.

[isoutamo]

It’s just like @PickleRick said. 
When splunk move buckets into frozen it wants that script or whatever you are using will return zero. After that it removed original bucket. It return value is something else then splunk try it again after some time.

[hazem]

@isoutamo @PickleRick 

you mean that Once logs transition to frozen storage, they are typically outside Splunk's control unless you have configured specific mechanisms to manage them.

 can I  handle frozen storage  using Automate Deletion of Old Frozen Data

Example: Delete Frozen Data Older Than 1 Year

#!/bin/bash

# Path to the frozen storage directory
FROZEN_DIR="/data/splunk_frozen"

# Log file for the operation
LOGFILE="/var/log/splunk_frozen_cleanup.log"

# Retention period in days (365 days = 1 year)
RETENTION_DAYS=365

# Find and delete directories older than the retention period

echo "$(date): Starting cleanup of frozen data in $FROZEN_DIR" >> "$LOGFILE"
find "$FROZEN_DIR" -type d -mtime +$RETENTION_DAYS -exec rm -rf {} \; -exec echo "$(date): Deleted {}" >> "$LOGFILE" \;

echo "$(date): Cleanup complete" >> "$LOGFILE"

[PickleRick]

Close, but there is no "unless" - Splunk will not keep track of your frozen buckets whatever you do. The last moment Splunk cares about your data is when it moves it to frozen. After that you're completely on your own. That's also why if you want to get that data back into Splunk you have to move the frozen buckets into the thawed directory and manually rebuild them.

So yes, you need to have some way to manually rotate frozen data. Your script is one of possible ways of handling that. Just be aware that 365 days from the move to frozen might happen not when you want, especially if you have some compliance-based requirements for data retention. It's probably better (but more troublesome) to base your retention script on timestamps from the bucket directory name.

And last but not least - if you have a cluster, each node manages its own buckets so you might end up with several copies of the same bucket.

[isoutamo]

And quite probably those copies have moved into froze totally different time, based on situation on individual indexer. Moving it is always initiated by indexer not master.