I am looking to move my splunk environment onto Amazon Web Services. I have read through a lot of the documentation regarding recommended hardware for my Splunk ec2 Instances. It seems there are two options with regards to backing up data and disaster recovery.
Use c4.??xlarge instances with EBS backed storage or d2.??xlarge instances with HDD based local storage and index replication between the instances.
Data loss prevention is extremely important to me and is my highest priority here. Can anyone give me any recommendations/ pros and cons of each method? Thanks!
We're actually evaluating the same thing. Here are the quick bullet points I've come up with so far:
If you're going to consider the d2's, you have to do so with index replication. Leverage the location awareness of the feature with AWS Availability Zones. (An indexer in each AZ configured such that a copy of each bucket exists in all of them) With the increased capacity of the d2 instances, it's much easier on the pocket to use replication.
The nice thing about using the d2's with replication is it lets you be more flexible in scaling your environment in response to load (Which may or may not matter depending on your particular use case) You can add/remove indexers from the cluster to scale up/down as long as you give the cluster opportunity to recover when pulling instances.
The cost differences can depend on the capacity you need and how you plan to buy it. The 70% discount on 1 year pre-paid d2 instances makes a BIG difference when comparing cost, too. So do the math on your particular needs. I've found this tool very helpful in determining what's needed. Splunk Sizing
One thing I will recommend is, even if you don't use replication, set up your indexers in a cluster from the start. It will make your life worlds easier if you decide to change your replication settings in the future, and makes configuration management easier.
So Finally what you went with? I am also looking at both EBS and d2. I am mainly concern with Performance not price. Which is best in terms of performance when replication is there.
We recently switched from EBS to d2's, and are very happy with the results so far. using d2.4xls with the ephemeral in a RAID0 gives us IOPS in the 7k range, and we can handle indexing throughput spikes of up to 7MB/s without appreciable queue saturation. Yes, EBS can get better IO, but the fact that you're throttled on throughput makes it hard to actually take advantage of it.
It's been about 4 months so far, and we haven't had an indexer fail yet. We have had one go temporarily unavailable for about 20 minutes, but there was no data loss. The cluster handled that without any real end-user impact.
All in all, I would have a hard time recommending EBS backed instances unless the install is small enough that running multiple d2s is overkill. You definitely want to have replication happening if you use them.
Thanks Emiller for quick response. We will have ~600GB per day data ingestion and 90 day retention and I am planning to use 4 indexers ( RF=3,SR=3) - 2 instance in each AZ(AZ1/AZ2).
Either use c4.4x large with IOPS SSD(30 day data) and Throughput optimized HDD (st1) for rest 60 day data else 4 instance of d2.4x large instance.
What do you think about this configuration?