Knowledge Management

Shared datamodels and CPU usage on indexers

PickleRick
SplunkTrust
SplunkTrust

Following https://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Sharedatamodelsummaries I set up sharing acceleration summaries between two search-head clusters.

I found guid of one of the clusters, set it up as a source_guid into a default stanza on the other cluster (first cluster uses CIM app and ES, the second one has just CIM app with datamodel settings migrated from first cluster).

So datamodel settings on the second cluster is  a subset of settings from the first cluster (I did a btool dump of dataset settings and compared them with vimdiff). On first cluster I have some addiional datamodels from ES app, the rest datasets is identical on both clusters (of course apart from the source_guid attribute).

As far as I understand the article, it should just work.

But as far as I add the CIM app (define the datamodels) on the second cluster, it starts killing my indexers.

I have 20CPU nodes with 64G of RAM and their load is typicaly around 6-7 and memory usage doesn't exceed 40G. Since the added the CIM app, load is doesn't fall below 40(!) and sometimes jumps to around 45 and the RAM is all used (I  even get oom-killers every half an hour or so).

The monitoring console shows that most resources (by a great margin) is used by datamodel acceleration.

And the top memory-consuming searches are various instances of _ACCELERATE_DM_Splunk_SA_CIM_Network_Traffic_ACCELERATE_

I don't understand however:

1) Why doesn't splunk just use the data I pointed it to? It seems to be "rebuilding" the summaries (and yes, I have a lot of network data, so it makes sense)

2) Why does it spawn the consecutive acceleration searches when the old ones didn't complete yet?

Labels (1)
0 Karma
1 Solution

PickleRick
SplunkTrust
SplunkTrust

It turned out - after a long and painfull debugging with the support team 😉 that you can't set the acceleration sharing in the [default] stanza in another app and let the datamodels inherit it. Even though btool shows the data as applied into each datamodel, the summary sharing doesn't work. You have to specify the source guid in the same app as on the source shcluster and within the configuration stanza of every single datamodel you want to share the summaries for.

View solution in original post

PickleRick
SplunkTrust
SplunkTrust

It turned out - after a long and painfull debugging with the support team 😉 that you can't set the acceleration sharing in the [default] stanza in another app and let the datamodels inherit it. Even though btool shows the data as applied into each datamodel, the summary sharing doesn't work. You have to specify the source guid in the same app as on the source shcluster and within the configuration stanza of every single datamodel you want to share the summaries for.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...