Are there any issues with our plan for scaling our multisite indexer cluster in an AWS environment?

[agoebel]

I mostly have a multisite cluster set up properly. The only remaining issue is we were hoping to scale up to 3 indexers when our system is under load (events, tests, etc) and scale back down to just a single indexer when there is a low load. I have the AWS autoscale group working, just wondering if there is an issue with this plan; if searches will fail if I don't reconfigure the master as well or any other gotchas people know about.

[mahamed_splunk]

You can add and remove indexers anytime, but the question will you also increase the RF/SF values when you add indexers ? For eg, will you change RF=3 when running with 3 indexers and change it back to RF=1 with 1 indexer ?

[agoebel]

Yes, the idea is the issue isn't adding or removing RF/SF's so much as being able to cleanly say change how the system is running for the next day or so. If this is as simple as making sure to change the RF/SF settings and restarting the master that would be convenient.

[mahamed_splunk]

Well, if all of your data is with 1 indexer and you 2 additional indexers to the mix, the newly added indexers are not going to help in handling the load right away, the reason is all your data is with 1 indexer so all the search queries will hit the same indexer. The better option would be to have more indexers, spread the data among the indexers evenly so that the search queries will be processed by all indexers

[agoebel]

The issue is one of cost. We are running in AWS and for the most part only need to handle a constant stream of logs for very short windows. It doesn't make sense to leave the compute power up constantly for the trickle of logs that come in and instead need to scale up and down based on the load of logs coming through. I don't have the exact numbers but its is the difference between a 10's of MB and 10's of GB's and something like 10 times more machines running.

As an aside, I find it hard to believe we are the only company that has this scenario.

[mahamed_splunk]

The alternative is, when you add new indexers, increase your RF values this will spread the data among the existing indexers and help you in handling the search workload.