coldtofrozenscript in a Clustered Enviroment

[Rhidian]

Does anyone have an example of a coldtofrozenscript to be deployed in a clustered enviorment, I'm weary of having duplicate buckets etc?

[dural_yyz]

Let Splunk manage the buckets for you.

https://docs.splunk.com/Documentation/Splunk/9.3.0/Indexer/Automatearchiving

Currently your script doesn't seem to have any filter on cold and would appear to copy all files so is this a one time execution as this on a cron would seem to copy over and over again creating a storage issue.

Benefits of Splunk Management

1) Frozen reduces storage as the raw data in compressed format is stored, the IDX files are stripped away

2) Easy methods to reintroduce Frozen data back to thawed

3) Expands automatically if you have IDX clustering

4) Your not duplicating storage of Cold qualified time spans in a manual frozen folder

5) Folder management and creation is automatic

[Rhidian]

This is to be used as coldToFrozenScript essentially I'm trying to avoid having multiple copies of my bucket. As per the guidance:

"In an Indexer cluster, each individual peer node rolls its buckets to frozen, in the same way that a non-clustered indexer does; that is, based on its own set of configurations. Because all peers in a cluster should be configured identically, all copies of a bucket should roll to frozen at approximately the same time."

[gcusello]

Hi @Rhidian ,

you have to move only data indexed on that Indexer and not replicated data.

You can distinguish them by the folder name:

• locally indexed data have folder names that start with db_
• replicated data data have folder names that start with rt_

In this way, you can create your own script.

Obviously only folders in the cold folder of each index.

Ciao.

Giuseppe

[Rhidian]

Thanks. So somthing simple like this should work?

#!/bin/bash

# coldToFrozen script for Splunk

# Arguments:
# $1 - Path to the cold bucket
# $2 - Path to the frozen bucket

COLD_BUCKET_PATH="$1"
FROZEN_BUCKET_PATH="$2"

echo "Starting coldToFrozen transition..."

# Log paths for debugging
echo "Cold Bucket Path: $COLD_BUCKET_PATH"
echo "Frozen Bucket Path: $FROZEN_BUCKET_PATH"

# Ensure paths are not empty
if [ -z "$COLD_BUCKET_PATH" ] || [ -z "$FROZEN_BUCKET_PATH" ]; then
    echo "Error: Cold or Frozen bucket path is not provided."
    exit 1
fi

# Check if the cold bucket directory exists
if [ ! -d "$COLD_BUCKET_PATH" ]; then
    echo "Error: Cold bucket path does not exist."
    exit 1
fi

# Create frozen bucket directory if it does not exist
if [ ! -d "$FROZEN_BUCKET_PATH" ]; then
    echo "Creating frozen bucket directory at: $FROZEN_BUCKET_PATH"
    mkdir -p "$FROZEN_BUCKET_PATH"
fi

# Move files prefixed with 'db_' from cold to frozen
echo "Moving 'db_' files from cold to frozen..."
for file in "$COLD_BUCKET_PATH"/db_*; do
    if [ -f "$file" ]; then
        mv "$file" "$FROZEN_BUCKET_PATH"
        if [ $? -ne 0 ]; then
            echo "Error: Failed to move file $file to frozen storage."
            exit 1
        fi
    fi
done

echo "Data successfully moved to frozen storage."

exit 0

[gcusello]

Hi @Rhidian ,

it seems correct, even if I'm not a script developer, you have only to test it.

Ciao.

Giuseppe