Can we use instances recommended by AWS compute optimizer for hosting Splunk in AWS

[AmithB]

We have a few instances hosted in AWS that are extremely underutilized (single digit avg. cpu% for the 3 months.

The AWS compute optimizer has recommended the following changes to the instances

Current Instance Type | Recommended Instance Type

c4.4xLarge  | r6i.xlarge

c4.8xlarge | r6i.2xlarge and  r6i.xlarge

c5.2xLarge | r6i.large,  r6i.xlarge,  t3.medium,  t3.small

c5.4xlarge | r6i.2xlarge

c5.9xlarge | r6i.4xlarge

c5.xLarge | r6i.large

t3.medium |  t3.large

t3.micro | t3.medium

We noticed that most of the recommendations are about replacing 'compute-optimized' instances with new-gen 'mem-optimized' intances. This also reduced the CPU cores. 

Question - can we consider and replace the instances based on the recommendations.

[AmithB]

Is there a reference document that helps us identify the number of CPU cores vs. concurrent searches that can be run.

We want to take this back to security folks to see if there is an opportunity to optimize the current underutilized instances (single digit CPU%), and thereby reduce costs.

[richgalloway]

I'm not aware of such a document, but you can find the number of concurrent searches using the Monitoring Console (MC).  The maximum number of concurrent systems is 6+<numCPUs>.  That formula can be modified using limits.conf settings, but is good for most environments.

If you see errors in the MC about searches being skipped because the maximum number of concurrent searches has been reached then you are not under-utilizing your server.  Try re-distributing your scheduled searches so fewer are running at the same time.  After that, if you still see the error then you are over-using the server and need more CPUs (or fewer searches).

If there are times when the server is not running any searches, then you are under-using it at those times.  The CPUs need to be available for times when searches run, however.

Perhaps you need lower-powered CPUs rather than fewer CPUs.

[richgalloway]

Yes, you can.  Whether you should or not is a different (and better) question.

CPU utilization is not a good measure of how well Splunk is using a VM.  Recall that the number of concurrent searches Splunk can run is based on how many CPUs are available.  Reduce that number and you reduce the number of searches you can run.

If your maximum concurrent searches is less than the number of CPUs available then a smaller VM might make sense; otherwise, look for other instance options.