Knowledge Management

Shared datamodels and CPU usage on indexers

PickleRick
SplunkTrust
SplunkTrust

Following https://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Sharedatamodelsummaries I set up sharing acceleration summaries between two search-head clusters.

I found guid of one of the clusters, set it up as a source_guid into a default stanza on the other cluster (first cluster uses CIM app and ES, the second one has just CIM app with datamodel settings migrated from first cluster).

So datamodel settings on the second cluster is  a subset of settings from the first cluster (I did a btool dump of dataset settings and compared them with vimdiff). On first cluster I have some addiional datamodels from ES app, the rest datasets is identical on both clusters (of course apart from the source_guid attribute).

As far as I understand the article, it should just work.

But as far as I add the CIM app (define the datamodels) on the second cluster, it starts killing my indexers.

I have 20CPU nodes with 64G of RAM and their load is typicaly around 6-7 and memory usage doesn't exceed 40G. Since the added the CIM app, load is doesn't fall below 40(!) and sometimes jumps to around 45 and the RAM is all used (I  even get oom-killers every half an hour or so).

The monitoring console shows that most resources (by a great margin) is used by datamodel acceleration.

And the top memory-consuming searches are various instances of _ACCELERATE_DM_Splunk_SA_CIM_Network_Traffic_ACCELERATE_

I don't understand however:

1) Why doesn't splunk just use the data I pointed it to? It seems to be "rebuilding" the summaries (and yes, I have a lot of network data, so it makes sense)

2) Why does it spawn the consecutive acceleration searches when the old ones didn't complete yet?

Labels (1)
0 Karma
1 Solution

PickleRick
SplunkTrust
SplunkTrust

It turned out - after a long and painfull debugging with the support team 😉 that you can't set the acceleration sharing in the [default] stanza in another app and let the datamodels inherit it. Even though btool shows the data as applied into each datamodel, the summary sharing doesn't work. You have to specify the source guid in the same app as on the source shcluster and within the configuration stanza of every single datamodel you want to share the summaries for.

View solution in original post

PickleRick
SplunkTrust
SplunkTrust

It turned out - after a long and painfull debugging with the support team 😉 that you can't set the acceleration sharing in the [default] stanza in another app and let the datamodels inherit it. Even though btool shows the data as applied into each datamodel, the summary sharing doesn't work. You have to specify the source guid in the same app as on the source shcluster and within the configuration stanza of every single datamodel you want to share the summaries for.

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...