What about archiving/offline storage?

[jones4bob]

What is the recommended way to export/archive a large amount of historical data for retention or offline storage? I would like to clean old data from splunk but still maintain our regulatory requirements on retention.

[dcroteau · Accepted solution]

The coldToFrozenScript (also specified in indexes.conf) runs just before the frozen data is erased. The default script simply writes the name of the directory being erased to the log file $SPLUNK_HOME/var/log/splunk/splunkd_stdout.log. If you want to archive frozen data rather than delete it, you'll need to substitute your own archiving script.

To substitute your own script, add the following stanza to $SPLUNK_HOME/etc/system/local/indexes.conf:

[] coldToFrozenScript =

Note the following:

* <index> specifies which index to archive.
* <script> specifies the archiving script.
      o Define the <$script> path relative to $SPLUNK_HOME/bin. The script needs to be located in that directory or a subdirectory. 

Splunk ships with two archiving scripts that you can modify (or you can create your own):

* compressedExport.sh: Export with tsidx files compressed as gz.
* flatfileExport.sh: Export as a flat text file (not recommended for current performance and resource issues -- it can take a long time, and use a lot of ram, 2-3GB, while running).