Send Splunk Archive Logs to Ceph S3

[splunkuser109]

How can we automatically send frozen/archived splunk logs from the indexers over to a Ceph S3 bucket using the indexers.conf file on the indexers? 

[richgalloway]

Yes, sort of.  Use indexers.conf to specify a coldToFrozenScript.  That script, which you must write, will copy the archived buckets to Ceph.  See https://docs.splunk.com/Documentation/Splunk/8.1.3/Indexer/Automatearchiving and https://docs.splunk.com/Documentation/Splunk/8.1.3/Admin/Indexesconf#PER_INDEX_OPTIONS

[splunkuser109]

• Hmm so it doesn’t look like there’s an easy way for us to automatically copy over the frozen logs directly to the ceph s3 bucket? 
  
  

Do you have any ideas on how we can write a script to copy over frozen buckets over to ceph s3 buckets? 

[richgalloway]

Personally, I like Visual Studio Code, but notepad++ is good, too.  😀

[splunkuser109]

hahaha. Can remotePath (ceph s3 bucket) not be used to store the cold or frozen buckets/logs? 

https://docs.splunk.com/Documentation/Splunk/8.1.0/Admin/Indexesconf

[richgalloway]

remotePath is used for warm/cold buckets.  This is part of the SmartStore feature.  Frozen buckets are different and are not stored by SmarStore.

There is an example coldToFrozenScript in $SPLUNK_HOME/bin.