Knowledge Management

Shared datamodels and CPU usage on indexers

PickleRick
Ultra Champion

Following https://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Sharedatamodelsummaries I set up sharing acceleration summaries between two search-head clusters.

I found guid of one of the clusters, set it up as a source_guid into a default stanza on the other cluster (first cluster uses CIM app and ES, the second one has just CIM app with datamodel settings migrated from first cluster).

So datamodel settings on the second cluster is  a subset of settings from the first cluster (I did a btool dump of dataset settings and compared them with vimdiff). On first cluster I have some addiional datamodels from ES app, the rest datasets is identical on both clusters (of course apart from the source_guid attribute).

As far as I understand the article, it should just work.

But as far as I add the CIM app (define the datamodels) on the second cluster, it starts killing my indexers.

I have 20CPU nodes with 64G of RAM and their load is typicaly around 6-7 and memory usage doesn't exceed 40G. Since the added the CIM app, load is doesn't fall below 40(!) and sometimes jumps to around 45 and the RAM is all used (I  even get oom-killers every half an hour or so).

The monitoring console shows that most resources (by a great margin) is used by datamodel acceleration.

And the top memory-consuming searches are various instances of _ACCELERATE_DM_Splunk_SA_CIM_Network_Traffic_ACCELERATE_

I don't understand however:

1) Why doesn't splunk just use the data I pointed it to? It seems to be "rebuilding" the summaries (and yes, I have a lot of network data, so it makes sense)

2) Why does it spawn the consecutive acceleration searches when the old ones didn't complete yet?

Labels (1)
0 Karma
1 Solution

PickleRick
Ultra Champion

It turned out - after a long and painfull debugging with the support team 😉 that you can't set the acceleration sharing in the [default] stanza in another app and let the datamodels inherit it. Even though btool shows the data as applied into each datamodel, the summary sharing doesn't work. You have to specify the source guid in the same app as on the source shcluster and within the configuration stanza of every single datamodel you want to share the summaries for.

View solution in original post

0 Karma

PickleRick
Ultra Champion

It turned out - after a long and painfull debugging with the support team 😉 that you can't set the acceleration sharing in the [default] stanza in another app and let the datamodels inherit it. Even though btool shows the data as applied into each datamodel, the summary sharing doesn't work. You have to specify the source guid in the same app as on the source shcluster and within the configuration stanza of every single datamodel you want to share the summaries for.

0 Karma
Get Updates on the Splunk Community!

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...

Security Highlights | January 2023 Newsletter

January 2023 Splunk Security Essentials (SSE) 3.7.0 ReleaseThe free Splunk Security Essentials (SSE) 3.7.0 app ...

Platform Highlights | January 2023 Newsletter

 January 2023Peace on Earth and Peace of Mind With Business ResilienceAll organizations can start the new year ...