How to make Splunk effectively use hardwares?

[VatsalJagani]

Splunk hardware recommendation is as follows:

• Normal Instance - SH and IDXs - 12 Core/16 GB
• Enterprise Security - SH and IDX - 16 Core/32 GB

But, when we see resource utilization is quite low. Like CPU and Memory utilization is quite less around 30-50%.

Can we make Splunk to really use the resources? How do you suggest to use the below parameters in limits.conf file?

• max_mem_usage_mb - Provides a limitation to the amount of RAM, in megabytes (MB), a batch of events or results will use in the memory of a search process.
• base_max_searches - A constant to add to the maximum number of searches, computed as a multiplier of the CPUs.
• max_searches_per_cpu - The maximum number of concurrent historical searches for each CPU.

Any recommendations or suggestions around this?

[thambisetty]

How much volume are you indexing per day?

How many users you have?

How many scheduled searches you have?

[VatsalJagani]

50GB/day

Around 8-10 regular users.

About searches - We are using Windows Infrastructure and PaloAlto as the main App in non-ES SH.
- And on ES we have around 6-8 correlation searches that we have enabled.

[isoutamo]

I think that there is no reason to modify those yet. Your load for nodes haven’t so big yet, so it’s obvious that there are additional recourses waiting that your load will increasing.

Just add needed monitoring (what ever monitoring system you already have in your environment). Also you could add needed/wanted MC alerts  you should also regularly look what happens in your environment with MC, then it’s easier react when there will be abnormal usage.

r. Ismo

[isoutamo]

One question: Have you performance issue or why you want to modify those values? And if you have performance issues do you know why?

r. Ismo

[VatsalJagani]

We usually have warnings of "searches delayed and searches skipped" even though DMC resource monitoring charts show very little CPU and Memory utilization.