Hi Everyone,
Our environment consists of an indexer cluster and independent SHs. ES runs on a single SH. We are seeing memory usage spikes on indexer at certain times of the day/night. There is no consistency or pattern to this. Resource usage drops after a few hour usually without much intervention. Sometimes a peer is considered "down" when there is excessive memory and cpu usage on that peer. When this happens, the cluster tries to recover which causes a lot of unnecessary "bucket fixup". We have not upgraded the servers recently or updated ES. I can provide more details based on your questions. Here are a few observations:
1. When the memory spikes on indexers, there are multiple executions of the datamodel accelerations running during the same instant (referring to the _time). Count is 2 or 3. Max concurrency for datamodels is set to 3. At other times (when memory usage is low), only 1 execution is seen. Please see below for clarification of the count I am referring to:
2. On some days, search concurrency in the cluster was too high (over 200). Am working on reducing the number of concurrent searches allowed on SH and available to scheduled searches. But this is also not consistent. For example, we did not have that many concurrent users or searches in the environment but we still had high memory usage across indexers.
Any help or insight would be appreciated. Working with support as well but it's unclear why the datamodels suddenly push the indexers to use over 80% of memory. Our machines are over-provisioned for the most part.
For example, the acceleration. that normally takes less than 3G would suddenly take over 5G or 9G of memory
Thanks!
