Monitoring Splunk

high usage of physical memory on indexers

Communicator

Hello,

We are having high usage of memory usage on all of our indexers and most of it is cached memory. can we clear it or will it have any impact if we do that. Please let me know if anyone have idea about this.

Thanks.

0 Karma

Builder

if you are running on linux, please check if the THP is disable, this can cause performance issues on system. Check this link -> https://docs.splunk.com/Documentation/Splunk/7.3.1/ReleaseNotes/SplunkandTHP
Also use the management console to check the indexing activity, select the option indexing/performance/instance and check the indexing queue pipeline to see how the indexing activity is going. If you the CPU is not being too much used, you can create a 2 pipeline to assist the indexing to output more data to the reports ->https://docs.splunk.com/Documentation/Splunk/7.3.1/Indexer/Pipelinesets
Check if exist a lot of skipping searches and it can also cause problems to the servers, please check this answer it can help you -> https://answers.splunk.com/answers/565482/how-to-resolve-skipped-scheduled-searches.html
Validade your hardware capacity and also considering to add more indexers on your indexers tier because it will give you more processing capacity. I hope this can help you.

Communicator

Thanks for the response. I was more looking to an answer which @jtacy provided in the comments.

0 Karma

SplunkTrust
SplunkTrust

How high is "high"? How much memory do your indexers have installed? Are they running a lot of searches or accelerations?

---
If this reply helps you, an upvote would be appreciated.
0 Karma

Communicator

we have around 500 gb and the total usage is aroung 460 gb. in that 460 gb 440gb is cached. yes, we have many scheduled searches running.

0 Karma

Builder

That's great! The more data is cached in memory, the better your frequently-run searches will perform because they won't have to retrieve as much data from disk. Yes, there will be impact if you clear the cache: your searches will be slower until the cache fills up again (which probably won't take long).

Is there a downside to a huge cache? In certain cases, the answer may actually be yes:
https://unix.stackexchange.com/questions/253816/restrict-size-of-buffer-cache-in-linux
Sounds like if you have an application that suddenly needs large amounts of memory, it may take longer to allocate memory that was used for cache than free memory. This is a pretty unique case and I can't imagine a situation where an indexer would be better off with more free memory than cache.

In short, the main reason to load up indexers with memory is to support more cache. Unless you're trying to solve a specific problem as in the above link, in almost all cases it's probably best to leave the cache alone.

Is there a specific performance problem you're trying to address?

Communicator

Thanks for the response. we are not doing any specific performance but we thought the usage is high and the cached is the one which is using most of it and thought we should clear that but as you said it is better to leave it same like that instead of clearing it. The usage is not increasing rapidly and sometimes going down as well.

0 Karma