Getting Data In

Help with Archiving Frozen data

Builder

Hi,

We have multi site indexer clustering with 2 sites, 3 indexers in each site with RF-3 and SF-2. Each indexer has its only drive for archives both rb_ and db_ buckets. Is there a way to archive together on a single standalone Indexer without archiving duplicate data? Please advise the best practice to achieve it.

Tags (3)
0 Karma

SplunkTrust
SplunkTrust

@kiran331,
the challenge is described well in docs:
http://docs.splunk.com/Documentation/Splunk/7.0.1/Indexer/Automatearchiving#Data_archiving_and_index...
not sure if it answers you needs but here is a trick. lets assume you either have another indexer (doesnt matter in which site) or, you can take one indexer out (again doesnt matter which site) and create a third "cluster" with 1 indexer call it site 3
now you can setup replication to make sure that this single indexer receives a copy of all the data in server.conf
read here:
https://docs.splunk.com/Documentation/Splunk/7.0.1/Indexer/Sitereplicationfactor#Examples
now, you have an archiving site consists of one indexer... single point of failure, but you can easily carve another "site" and set the replication rules to have a copy on "archiving site1" and "archiving site2"

hope it helps

0 Karma

Path Finder

Thanks adonio, currently all the data is going to site1 and replicating 1 copy to site 2. I think single indexer with site3 will be best option for us, but it if i move all frozen db_ buckets from site1 to a disk, will db_* buckets have replicated data too?

server.conf:
site_replication_factor = origin:2,site1:1,site2:1,total:3
site_search_factor = origin:1,site1:1,site2:1,total:2

0 Karma

SplunkTrust
SplunkTrust

keep buckets where they belong, they can only be restored at the indexer that they were created.
my suggestion above, is for moving forward.
your server.conf looks would have also site3=1 (archive indexer) for replication_factor that will ensure that you have a replicated copy in that indexer.
once applied, verify you have all replicas in site3 and modify you archiving procedure to freeze data only from site3. other data can age out safely as you supposed to have a replica and a frozen copy on site 3.
please let us know how it worked for you

0 Karma

Builder

Thanks Adonio, I have a question, will the all archives from a site with 1 replicated copy with 3 indexers and archives from 1 indexer site with 1 replicated be same (in terms of disk space)?

0 Karma

SplunkTrust
SplunkTrust

@kiran331,
hope i understand your question correctly.
when you "force" a copy on a site that has 1 indexer it means that regardless of which site the original copy is "landed" on, it will replicate a copy to that site (with single indexer).
setting an archiving (freezing) policy on this indexer will assure that you will have exactly 1 copy of all data on this indexers frozen data path.
2 challenges here are:
a. you will be able to thaw (recover) data only to this indexer.
b. you will probably need more storage on this indexer to accommodate retention.

hope it answers it

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!