topic Re: Reducing rentention costs, archiving frozen buckets, running multpiple instances in All Apps and Add-ons

Reducing rentention costs, archiving frozen buckets, running multpiple instances

sbsbb — Wed, 17 Apr 2013 11:09:56 GMT

Because the SAN Space is pretty expensive, we are only keeping the Data in Splunk 2 months.

Is it possible to have
- One instance from Splunk on the SAN for normal search (first 2 months)
- One instance on old Hardware, that reuse frozen buckets from the first instance (from 2 until 12 month old)
- The first instance beeing able to search in the two instances...

Or would it be possible to move all frozen bucket from the culstered indexer, to a "slow" indexer withou SAN ? (That would be my favorit solution, if possible)

I guess the drawback from shuttl is that I can't only search on the messages I want to see, I have to reload all the timerange needed in splunk ?

Re: Reducing rentention costs, archiving frozen buckets, running multpiple instances

alacercogitatus — Wed, 17 Apr 2013 12:27:30 GMT

For each index, you can specify different locations for the buckets. You can set the hot/warm buckets to the SAN storage, and the cold/frozen buckets to the "slow" storage.

Indexes.conf:

[volume:fastSan] path = /path/to/fast/san

[volume:slowSan] path = /path/to/slow/san

[myindex] homePath = volume:fastSan/myindex/db coldPath = volume:slowSan/myindex/colddb coldToFrozenScript = /MUST HAVE THIS TO MOVE THE DATA

Re: Reducing rentention costs, archiving frozen buckets, running multpiple instances

sbsbb — Wed, 17 Apr 2013 12:53:35 GMT

In the Splunk doc, there is somewhere a documentation that the warm and cold buket have to be on a medium with similar characteristic (not slower for cold...)

So it would be better to move data from a fastIndex to a slow index ... ?

Re: Reducing rentention costs, archiving frozen buckets, running multpiple instances

alacercogitatus — Wed, 17 Apr 2013 12:57:31 GMT

See here: http://docs.splunk.com/Documentation/Splunk/5.0.2/Indexer/Usemultiplepartitionsforindexdata. While they recommend keeping it all on one file system, you can split it up. UNLESS you are using clustering. Then you should keep hot/warm/cold on fast SAN.

Re: Reducing rentention costs, archiving frozen buckets, running multpiple instances

sbsbb — Thu, 18 Apr 2013 06:18:42 GMT

Thats my case... so what are the alternatives ?