Data Model Accelerations are very slow

[gcusello]

Hi at all,
I have an issue on Data Models accelerations: the run times of each accelerations are too high to use DMs in my Correlation Searches: more than 2000 seconds for each run.

I have six IDXs with 24 CPUs (only partially used: less that 50%) and storage with 1500 IOPS, so the infrastructure shouldn't be the issue.

Six Indexers should be sufficient to index and search 1TB/day of data, so this shouldn't be the issue.

I have around 1 TB/day of data distributed in more than 30 indexes and I listed these indexes in the CIM macro, so this shouldn't be the issue.

Where could I search the issue?

Now I'm trying with some parameters: I enabled "Poll Buckets For Data To Summarize" and I disabled "Automatic Rebuilds".

Is there something else in the DM structure that could be critical?

Thank you for your help.
Ciao. 
Giuseppe

[PickleRick]

We talked about this on Slack. 🙂

Six indexers might or might not be sufficient for 1TB/day. Depends on the search load.

Remember about two things:

1. Acceleration searches are quite heavy. They plow through heaps of data to create DAS (which they write additionally stressing I/O).

2. Acceleration searches are at the far end of the priority queue when it comes to scheduling searches.

So your acceleration searches might simply be running when there is already a whole lot of other things going in your system.

That's one thing.

Another thing is that CIM datamodels rely on tags and eventtypes which on the other hand rely on the raw data and how the fields are extracted. You might simply be having just "heavy" sourcetypes.

You could try to dry-run an acceleration search as an ad-hoc search (get it with the acceleration_search option for the datamodel command) and inspect the job to see where it spends most of its time.

It doesn't have to be directly tied to the datamodel or acceleration settings themselves (although it still might).

Of course, you could try to run more shorter searches as was already suggested (and what is recommended in  https://help.splunk.com/en/splunk-enterprise/manage-knowledge-objects/knowledge-management-manual/9....) but this will probably not lower your i/o load.

[gcusello]

Hi @PickleRick ,

I opened this question also on Community to share the problem and (I hope) the solution also in Community than in Slack.

Anyway, I am doing some tuning activity on the Indexers and I had some good result:

we configured:

• On indexers:
  • parallelIngestionPipelines: from 4 to 2
• On Search Heads, for each Data Model:
  • backfill range: 1 day,
  • max summarization search time: 1200 seconds,
  • Maximum Concurrent Summarization Searches: 4
  • Poll Buckets For Data To Summarize: unflagged
  • Automatic Rebuilds: unflagged

all these actions reduced the acceleration run times values from 3600 to 7-800 second for Authentication.

Now I'd lite to try to have better results:

do you think that passing parallelIngestionPipelines from 2 to 1 could reduce the acceleration run_rime value without creating indexing issues (for the moment we have queues=0 on all Indexers and all queues)?

Are there there other settings that I could try (remembering that this is a production system)?

Ciao and thanks

Giuseppe

[PickleRick]

As usual - it depends. The more parallel indexing pipelines you have, the higher theoretical possible indexing throuhghput (but the grow isn't linear). But also you're "binding" more CPUs on your indexers. Remember that each pipeline uses 4-6CPUs.

It's always the balance between indexing performance and search performance. Reducing indexing performance (because that's what removing pipelines is) will leave you with more performance for searching but yes, if you're close to the edge, it might result in clogging the input.

Unfortunately, I don't know of a 100% sure way to tell whether you can drop one more pipeline. You can check the reports on indexing->Perormance: Advanced screen in your MC to see if you're loaded to the brim or not yet, but that's still only an "educated guess". It's usually the other way around - if you have spare CPUs, you add more pipelines. I don't recall ever removing pipelines in a busy production environment.

[PrewinThomas]

@gcusello 

Your resources looks pretty good.
Can you check your DM search constraints are using any broad search constraints and too large acceleration summary range enabled?
Too many High-Cardinality Fields in the DM?

Also can you check Data model audit dashboard can provide any further details for this

Regards,
Prewin
Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!

[gcusello]

Hi @PrewinThomas ,

thank you for your support: I'm breaking my head from too much time!

Data Model Audit dashboard doesn't give any additional information that all the enabled accelerations have too high run_times values.

About acceleration summary range: I enabled only two days, infact the DM dimensions are very low.

About DM constrains: I used the related macros to search only on the relevant indexes but they are many: e.g. in Authentication DM there are more than 30 Indexes.

About High Cardinality fields: I have many of them (as user, src, dest, etc...) but in the Authentication DM they are relevant and always present, so I cannot remove them.

I also optimized scheduling.

I suppose that I'd search in acceleration parametrs but, at the moment, without luck!

Ciao.

Giuseppe

[meetmshah]

Are you using the default acceleration parameters? If so, can you try having Max Concurrent Searches as 4 (instead of 3), Max Summarization Search time as 15 Mins (instead of 60), Lower the backfill range (if you are sure that there are no major historical events we need to take care about). I faced the similar issue for a large (40 TB+ a day) customer, and had to tweak those parameters for Network_Traffic and couple of other Data Models.

Reference Doc - https://help.splunk.com/en/splunk-enterprise/manage-knowledge-objects/knowledge-management-manual/9....

[gcusello]

Hi @meetmshah ,

yes I used default parameters and then I'm trying to modify some of them, without luck!

Now I will try your hints and I'll inform you.

Ciao.

Giuseppe

[gcusello]

Hi at all,

I noted that there are very many buckets, do you think that it could be useful to enlarge the buckets dimension from the default (750 MB) to a larger value (e.g.: 1500 MB)?

Even if the effect will be sensible in the future.

Ciao.

Giuseppe

[meetmshah]

No, bucket size should be fine - just need to play around with 3 DMA parameters only - 

1. Backfill Range

2. Max Concurrent Searches

3. Max Summarization Search Time

So, the strategy I suggested previously was - "build short, build often". Check the attached file.

The boxes in the Yellow are default - where with Max Concurrent Searches with 3 and Max Search Time as 3600, the 4th concurrent search will only start once 1st is completed (after 3600) - which is happening in your case. However, we can go with the other approach - in green - where with 4 concurrent searches and Max Search Time as 1200 - we can build the summaries faster - which will also respect the recent event faster.

Please let me if any questions (and how the configuration changes goes!)

[PrewinThomas]

@gcusello I was wondering if your summary range is 2 days why your earliest time and latest time have a gap of around 17 months.

Also can you run this and check if this is also slow
 | tstats summariesonly=true count from datamodel=Authentication by _time span=1h

[gcusello]

Hi @PrewinThomas ,

I configured a summary range of 2 days to reduce the summarization time, but data have a retention of 30 days and anyway searches are always on the last 10-15 minutes.

Ciao.

Giuseppe

[PrewinThomas]

Also if possible can you share your datamodels.conf for authentication dm.

[gcusello]

Hi @PrewinThomas ,

this is authentication DM stanza in datamodels.conf:

[Authentication]
acceleration = true
acceleration.earliest_time = -2d
acceleration.hunk.dfs_block_size = 0
acceleration.poll_buckets_until_maxtime = true
acceleration.schedule_priority = default
tags_whitelist = cleartext,cloud,default,insecure,multifactor,pci,privileged

Ciao.

Giuseppe

[PrewinThomas]

@gcusello 
acceleration.backfill_time = 12h
acceleration.max_time = 1800
acceleration.manual_rebuilds = true

Can you add this to your .conf and check how it's running now.