Alerting

Frozen bucket archival to s3 in splunk trial version from AWS Marketplace

rajiv_r
Explorer

HI all,

I am new in splunk admin and doing a poc on archiving the frozen bucket data to the s3 bucket. Can I directly provide the s3 Url in the splunk web under index setting or do i need to provide the archive script in the coldtofrozen directory.

Also while setting the archiving policy do I need to change only frozenTimePeriodInSecs or do I need to change both maxTotalDataSizeMB  and frozenTimePeriodInSecs.

Please excuse this silly question as i am new in Admin i am looking for the best practise :slightly_smiling_face:

 

0 Karma

tscroggins
Influencer

@rajiv_r 

Freezing buckets to something other than a mounted file system requires a custom script specified by the index's coldToFrozenScript parameter. The process is similar for both "classic" and SmartStore indexes:

https://docs.splunk.com/Documentation/Splunk/8.1.3/Indexer/Automatearchiving
https://docs.splunk.com/Documentation/Splunk/8.1.3/Indexer/SmartStoredataretention

Splunk provides an example script in $SPLUNK_HOME/bin/coldToFrozenExample.py.

If you're considering S3 and no other lower cost alternative like S3 Glacier, you may want to look at SmartStore (native S3 integration) instead.

There's nearly a decade of prior art on various ways to migrate frozen buckets to S3 but no "best practices" I'm aware of for implementing custom coldToFrozenScript scripts beyond the example provided by Splunk.

In effect, you'll be running an AWS CLI S3 command to copy <bucket>/rawdata/journal.gz to the S3 bucket of your choice.

0 Karma
Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...