topic Re: Datamodel Acceleration: How to reduce high memory usage? in Getting Data In

Datamodel Acceleration: How to reduce high memory usage?

olivier_ma — Mon, 01 Jul 2019 14:04:36 GMT

Hello,

I'm facing a high memory usage on all of the 3 indexers when I try to accelerate a datamodel, even for 1 day acceleration.
After investigation, high memory usage is due to searches run for datamodel acceleration.
Our indexers have 20Gb of RAM. The accelerated datamodel is about proxy logs (~60Gb / day) and it's an root-event based datamodel.

How can I debug this issue ?
Did I miss something on datamodel acceleration ?

Thanks

Re: Datamodel Acceleration: How to reduce high memory usage?

skalliger — Mon, 01 Jul 2019 17:19:41 GMT

What are your specs of the ES SH and the indexers? Are your DMAs completing in time?
Did you restrict the DM to the specific index(es)? You can also try increasing the number of instances assigned to the DMA, this can also be done via UI (or acceleration.max_concurrent in datamodels.conf).
Did you set skewing (called allow_skew ) for scheduled searches and DMA already?

If you still run into issues after doing some tuning, you might wanna contact support/PS about further improvements.

Skalli

Re: Datamodel Acceleration: How to reduce high memory usage?

olivier_ma — Tue, 02 Jul 2019 09:38:00 GMT

I don't have ES. Here is specs
1 SH (RHEL):
8 CPU, 12 Gb of RAM, Splunk 7.0.2
3 Indexers (RHEL)
16 CPU, 24 Gb of RAM, Splunk 7.0.2

This DMA didn't complete as indexers systems are killing splunk processes for high memory usage.
But other DMAs (on other DM) work perfectly.

All accelerated DMs are restricted to specific index.
I'm pretty afraid about increasing the number of concurrent acceleration as enabling this one (proxy DMA) is already causing high memory usage. Currently it is set to 1.

I didn't know about skew settings. I'll take a look.
Thanks for the hand. I'll let you know if it helps 🙂

Re: Datamodel Acceleration: How to reduce high memory usage?

skalliger — Tue, 02 Jul 2019 09:49:39 GMT

How much are you ingesting per day and what is the size of the mentioned DM?
If only the big DMA doesn't get completed in time right now, skewing could help but in the long term you might wanna add another indexer to distribute the load further.

Skalli

Re: Datamodel Acceleration: How to reduce high memory usage?

woodcock — Sat, 06 Jul 2019 03:05:19 GMT

Configure BOTH index= AND sourcetype= in your cim_<DMA>_index macros.
Configure tags whitelist.
Check out the known issues page of the release notes for your release and implement the workarounds listed for the DMA bugs for your version (there are some DOOZIES).
Get more RAM for your Search Heads and Indexers and RESERVE IT EXCLUSIVELY if in a VM.
Deploy more Indexers.