Getting Data In

Help with Archiving Frozen data

kiran331
Builder

Hi,

We have multi site indexer clustering with 2 sites, 3 indexers in each site with RF-3 and SF-2. Each indexer has its only drive for archives both rb_ and db_ buckets. Is there a way to archive together on a single standalone Indexer without archiving duplicate data? Please advise the best practice to achieve it.

Tags (3)
0 Karma

adonio
Ultra Champion

@kiran331,
the challenge is described well in docs:
http://docs.splunk.com/Documentation/Splunk/7.0.1/Indexer/Automatearchiving#Data_archiving_and_index...
not sure if it answers you needs but here is a trick. lets assume you either have another indexer (doesnt matter in which site) or, you can take one indexer out (again doesnt matter which site) and create a third "cluster" with 1 indexer call it site 3
now you can setup replication to make sure that this single indexer receives a copy of all the data in server.conf
read here:
https://docs.splunk.com/Documentation/Splunk/7.0.1/Indexer/Sitereplicationfactor#Examples
now, you have an archiving site consists of one indexer... single point of failure, but you can easily carve another "site" and set the replication rules to have a copy on "archiving site1" and "archiving site2"

hope it helps

0 Karma

knalla
Path Finder

Thanks adonio, currently all the data is going to site1 and replicating 1 copy to site 2. I think single indexer with site3 will be best option for us, but it if i move all frozen db_ buckets from site1 to a disk, will db_* buckets have replicated data too?

server.conf:
site_replication_factor = origin:2,site1:1,site2:1,total:3
site_search_factor = origin:1,site1:1,site2:1,total:2

0 Karma

adonio
Ultra Champion

keep buckets where they belong, they can only be restored at the indexer that they were created.
my suggestion above, is for moving forward.
your server.conf looks would have also site3=1 (archive indexer) for replication_factor that will ensure that you have a replicated copy in that indexer.
once applied, verify you have all replicas in site3 and modify you archiving procedure to freeze data only from site3. other data can age out safely as you supposed to have a replica and a frozen copy on site 3.
please let us know how it worked for you

0 Karma

kiran331
Builder

Thanks Adonio, I have a question, will the all archives from a site with 1 replicated copy with 3 indexers and archives from 1 indexer site with 1 replicated be same (in terms of disk space)?

0 Karma

adonio
Ultra Champion

@kiran331,
hope i understand your question correctly.
when you "force" a copy on a site that has 1 indexer it means that regardless of which site the original copy is "landed" on, it will replicate a copy to that site (with single indexer).
setting an archiving (freezing) policy on this indexer will assure that you will have exactly 1 copy of all data on this indexers frozen data path.
2 challenges here are:
a. you will be able to thaw (recover) data only to this indexer.
b. you will probably need more storage on this indexer to accommodate retention.

hope it answers it

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!