Knowledge Management

Optimizing Accelerated Data Models

danieladams
Engager

My company is heavily using CIM accelerated data models for our security monitoring. We are currently experiencing performance issues and we think that data model acceleration is contributing to them. The searches that accelerate these data models are consistently the top memory-using searches, they run for a long time, and they are often behind. Is there a way to optimize these searches so that they aren't as taxing on the system? Here's how we have the CIM macros set up:

 (index="index1" AND sourcetype="sourcetype1")
    OR (index="index2" AND (sourcetype="sourcetype2" OR sourcetype="sourcetype3")
    OR (index="index3" AND sourcetype="sourcetype4")

And so forth. Any suggestions? We are on version 7.0.4.

cstump_splunk
Splunk Employee
Splunk Employee

First off, if you have not upgraded to either the latest core version of 7.2, 7.3, or 8.0, I recommend you do so as there have been updates that optimize the acceleration process.

Secondly, you should check to see if your datamodels have high cardinality fields, i.e fields with a high number of unique events. This leads to additional memory consumption on the Search Peers.
One way I like to do this is by using:

|datamodel <DM_name> acceleration_search 
| stats estdc(*) 
| transpose

Running this for a short 15-minute window in the past will tell you if any of the fields in the DM have high cardinality. If possible, either filter out some of the values in these fields or remove them for the DM definition.

Get Updates on the Splunk Community!

Index This | Divide 100 by half. What do you get?

November 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Stay Connected: Your Guide to December Tech Talks, Office Hours, and Webinars!

❄️ Celebrate the season with our December lineup of Community Office Hours, Tech Talks, and Webinars! ...

Splunk and Fraud

Watch Now!Watch an insightful webinar where we delve into the innovative approaches to solving fraud using the ...