Splunk Enterprise Security

Splunk ES DMA - "Accelerate until maximum time"

splunk_zen
Builder

Need some clarification regarding enabling
"Accelerate until maximum time"

according to the docs
"When selected, runs the acceleration search until the maximum time is reached."

Does this mean that even if a DM acceleration takes say 2min, if the "Maximum time" is set to 3600 seconds, it will never consider it as finished until it reaches that value?

0 Karma

CyberSekure
Explorer

I've got a customer who is experiencing this same issue. As soon as I turn off the "Accelerate until maximum time" setting, the run times drop back to normal levels. They're on SplunkCloud 7.0.9.1 if that helps. Will raise with Splunk support as well.

0 Karma

CyberSekure
Explorer

Just had a look in datamodels.conf - I think this setting refers to:

acceleration.poll_buckets_until_maxtime = <bool>

When set to "true": All of the machines run for "max_time" (approximately).

0 Karma

warwicks
Explorer

No it is till complete or max_time.
Have a look at how long your DM acceleration searches are taking to complete with something like

| rest /services/search/jobs/ 
| search label="*"
| eval srchEarliestTime=strftime(searchEarliestTime,"%Y/%m/%d %H:%M:%S")
| eval srchLatestTime=strftime(searchLatestTime,"%Y/%m/%d %H:%M:%S")
| table id sid latestTime label author eai:acl.app dispatchState doneProgress isDone isFailed runDuration resultCount diskUsage srchEarliestTime srchLatestTime ttl

I haven't dived in to all those fields some like doneProgress and isDone seem to disagree occasionally but I need to read more about what they are trying to tell me. There are plenty of other fields of interest as well have a play with which ones are useful but runDuration seems useful.

0 Karma

splunk_zen
Builder

Thanks but that's not what I'm seeing.
I had two DMAs taking a scary 14000+ and 6000+ and seconds to compute and as soon as I disabled that parameter their processing came down to 300ish seconds (and are still showing up with 100% datamodel acceleration)

0 Karma

maraman_splunk
Splunk Employee
Splunk Employee

the summary search may be running for 1h but that doesnt mean it really use 1h ressources (especially if you haven't 1 hour of data to process). But it will take new data as it arrive without having to wait for starting another summary which is 1 of the use case of this mode I think in order to reduce avg and max dma acceleration lag.

0 Karma

splunk_zen
Builder

Thanks maraman, this had been raised by another team's solutions architect exactly because of the extremely avg and max dma acceleration times, those numbers didn't look good and they were using them against Splunk
I may have run into an unusual combination of factors but I was only getting about 3 or 4 runs per day for the slowest DM (runs were being deferred as it was already running) which was an indication something fishy was going on

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!