AWS deployment requirements

[mwdbhyat]

Hi guys,

Looking to deploy Splunk on AWS and curious how it translates compared to physical servers. I have around 3TB a day, 30 concurrent users(60 total users), running ES and planning to implement smartstore in a multisite cluster(1 region, 2 AZ). Roughly i am looking at:

1 x SH - M4.10xlarge - concern here being that docs say it supports up to 20 concurrent users for ES - we will have 30.
17 x IDX i3.8xlarge - would this be sufficient for indexing/search needs?

There is still alot more low level detail I need to gather so I understand this is hard to accurately suggest - more interested in seeing if my assumption on a loose outline is headed in the right direction. Any input would be appreciated!

Thanks!

[esix_splunk · Accepted solution]

As you mention, there are a lot of moving parts to this to consider. In general our AWS sizing guidance holds up. I would use the i3en instances over i3 at this time. The performance is better, and this is what Splunk Cloud currently deploys. Additionally, our sizing is more around concurrent searches and not concurrent users, event though they are connected. ( 1 user vs 1 user running 15 searches..)

For ES,  we still will say target 100 to 150gb/day per indexer. But that being said, on Splunk versions 7.2+, you should be able to tune and see numbers closer to 200 or even higher (depends heavily on your search workload, DMA etc.) 

With that being said and not knowing what your data sources and volumes are, and using 150 as the base line, 20 indexers should be the baseline target for deployment. That isn't a huge variance from your 17 number, that would be about 175gb/day per indexer. And again with tuning, that in general shouldnt be difficult to achieve.  

Regarding your search head, I'd look at the M5 series over the M4.. At this scale, I'm curious why you are not deploying a SHC. You get the benefit of HA/DR here, plus the capability to horizontally scale your search tier as you need to grow. Although I do like having one large beefy instance, you'll get better performance and scalability going with something like 3 x m5.8xlarge. Benefits here are numerous, and with ES I am going to assume this is for S/NOC which will have some HA requirements that a single instance will most likely not adhere to..

As always, a deployment at this scale has a lot of moving parts, and the above are just some general high level talking points without knowing the full depth of the environment. You definitely should involve your Splunk SE / Sales team and ask for sizing assistance on this. We have some internal tools that will help provide a more accurate picture of what you need.

[mwdbhyat]

Thanks for the detailed explanation! 

[nyc_jason]

Hello mwdbhyat,

There is a capacity planning guide for splunk:

 https://docs.splunk.com/Documentation/Splunk/8.0.4/Capacity/IntroductiontocapacityplanningforSplunkE...

Fot 3TB/day, you may want to consult your sales team and their SE.