Reducing rentention costs, archiving frozen buckets, running multpiple instances

[sbsbb]

Because the SAN Space is pretty expensive, we are only keeping the Data in Splunk 2 months.

Is it possible to have
- One instance from Splunk on the SAN for normal search (first 2 months)
- One instance on old Hardware, that reuse frozen buckets from the first instance (from 2 until 12 month old)
- The first instance beeing able to search in the two instances...

Or would it be possible to move all frozen bucket from the culstered indexer, to a "slow" indexer withou SAN ? (That would be my favorit solution, if possible)

I guess the drawback from shuttl is that I can't only search on the messages I want to see, I have to reload all the timerange needed in splunk ?

[alacercogitatus]

For each index, you can specify different locations for the buckets. You can set the hot/warm buckets to the SAN storage, and the cold/frozen buckets to the "slow" storage.

Indexes.conf:

[volume:fastSan]
path = /path/to/fast/san

[volume:slowSan]
path = /path/to/slow/san

[myindex]
homePath = volume:fastSan/myindex/db
coldPath = volume:slowSan/myindex/colddb
coldToFrozenScript = /MUST HAVE THIS TO MOVE THE DATA


[sbsbb]

Thats my case... so what are the alternatives ?

[alacercogitatus]

See here: http://docs.splunk.com/Documentation/Splunk/5.0.2/Indexer/Usemultiplepartitionsforindexdata. While they recommend keeping it all on one file system, you can split it up. UNLESS you are using clustering. Then you should keep hot/warm/cold on fast SAN.

[sbsbb]

In the Splunk doc, there is somewhere a documentation that the warm and cold buket have to be on a medium with similar characteristic (not slower for cold...)

So it would be better to move data from a fastIndex to a slow index ... ?