Knowledge Management

Optimizing Accelerated Data Models


My company is heavily using CIM accelerated data models for our security monitoring. We are currently experiencing performance issues and we think that data model acceleration is contributing to them. The searches that accelerate these data models are consistently the top memory-using searches, they run for a long time, and they are often behind. Is there a way to optimize these searches so that they aren't as taxing on the system? Here's how we have the CIM macros set up:

 (index="index1" AND sourcetype="sourcetype1")
    OR (index="index2" AND (sourcetype="sourcetype2" OR sourcetype="sourcetype3")
    OR (index="index3" AND sourcetype="sourcetype4")

And so forth. Any suggestions? We are on version 7.0.4.

Splunk Employee
Splunk Employee

First off, if you have not upgraded to either the latest core version of 7.2, 7.3, or 8.0, I recommend you do so as there have been updates that optimize the acceleration process.

Secondly, you should check to see if your datamodels have high cardinality fields, i.e fields with a high number of unique events. This leads to additional memory consumption on the Search Peers.
One way I like to do this is by using:

|datamodel <DM_name> acceleration_search 
| stats estdc(*) 
| transpose

Running this for a short 15-minute window in the past will tell you if any of the fields in the DM have high cardinality. If possible, either filter out some of the values in these fields or remove them for the DM definition.

Get Updates on the Splunk Community!

Dashboard Studio Challenge - Learn New Tricks, Showcase Your Skills, and Win Prizes!

Reimagine what you can do with your dashboards. Dashboard Studio is Splunk’s newest dashboard builder to ...

Introducing Edge Processor: Next Gen Data Transformation

We get it - not only can it take a lot of time, money and resources to get data into Splunk, but it also takes ...

Take the 2021 Splunk Career Survey for $50 in Amazon Cash

Help us learn about how Splunk has impacted your career by taking the 2021 Splunk Career Survey. Last year’s ...