Splunk buckets restoration from amazon s3 bucket on indexer cluster with auto balancing.

[deepashri_123]

Hi Splunkers,

I have a clustered environment on AWS infrastructure with 2 indexers 1 master and 2 search heads.
We have been taking backup of both the indexers on AWS s3 buckets on daily basis for high availability.
When both indexers are down and new instances are spin up,we are restoring the data on new indexers from s3 bucket.
My script restores buckets on only 1 indexer and i need to rebalance it. But for larger data this option is not feasible.

What can be the option to restore data on both the indexers with load balancing?
Also in-case of auto-scaling my indexers number might also increase.

[nickhills]

Are you backing up the data - or archiving it?

The recommended archiving approach in a cluster, is to configure an archive site, and use the cold2frozen process on that site to perform the archiving. This method means you only have 1 archive copy of the data.

If you are backing up your Indexers for DR/BCP reasons, you need to consider the type of disasters you are designing the solution for.
In a cluster you will (should) have more than 1 copy of each bucket. Therefore if one of your indexers fails, your other one can still service requests. Simply (ha) rebuild the old one, and let the cluster fix-up.

If your worried about a total site-loss, then you can either do what you are doing now, or locate another indexer in a different site.
(Maybe in another AWS region?)

If you suffer a site-loss, and your indexers are irretrievable (fire/theft etc.) then your best approach is to rebuild and restore both indexers in the cluster, or restore 1 copy of the data and re-balance. This will take some time, and there is no quick solution to this issue, however if "fast time to recovery" is more important than "cost" I would go for an indexer located in an additional site.

You can have it:
-reliable
-fast
-cheap

pick ONLY 2.

On your second point about auto scaling - No. Don't do this!
You are heading for a world of trouble if you are thinking about scaling Splunk indexers in and out. The current Splunk architecture is not designed to accommodate this approach. I suggest a call with your Splunk account manager to discuss options if you are considering this.

[deepashri_123]

Hey nickhillscpl,

We are already doing multisite clustering, so site loss is already considered. But we are worried that if we have indexer loss on both sites which is a rare scenario ,since its a banking application we need to be sure that there is no data loss and hence the backup in s3.
Restoring 1 copy of data and rebalancing is not an option if the size of data is more than the indexer size. Thus looking for restoration script that would auto balance data on my indexers. And autoscaling has to be considered since the data stored on my s3 bucket can be more than my indexer size.

Thanks for the response!!!