Is there a general area on my SPLUNK server that contains all the indexed data that has been backed up from all my forwarders? I know I can script a save and clear of the data but is it possible for SPLUNK to automate this and then have a script that pulls the backed up data?
You'll want to read this first to understand our data lifecycle and your options. Let us know if you have questions after looking at this.
http://docs.splunk.com/Documentation/Splunk/6.0/Indexer/Backupindexeddata
The indexed data is located in splunk/var/lib/splunk/db unless you have configured it to go elsewhere in your indexes.conf files.
Right now, Splunk does not have a means of backing up the indexed data. If you wrote a script to make copies of the data it would be best to run it manually, outside of Splunk when Splunk is stopped. I can think of a few reasons for this, but the primary one is you can't (should not) backup Hot buckets. Hot buckets roll to warm buckets when Splunk is stopped, which can be backed up.
You'll want to read this first to understand our data lifecycle and your options. Let us know if you have questions after looking at this.
http://docs.splunk.com/Documentation/Splunk/6.0/Indexer/Backupindexeddata