topic Re: Data model acceleration not working in Reporting

Data model acceleration not working

sarit_s — Mon, 08 Jun 2020 23:57:17 GMT

Hello
Im running splunk data model acceleration
And it stopped working.
It is stuck in skipping and nothing happens
With “summariesonly=true” i get no results but if i set it to false i get results
Also, ive created new one event base and its working
The first one was search base
I couldn’t find any errors in the logs
Any suggestions?

Re: Data model acceleration not working

nickhills — Thu, 20 Feb 2020 18:41:20 GMT

If you have skipping searches, it would suggest that your searchhead(s) are struggling under search load.
Is this a stand alone SH or a Search Head Cluster?

The reason a small data model might complete is because the search to accelerate it is completing quickly, whereas longer searches are being deferred or skipped.

You need to identify is if this has started "out of the blue" or has been a problem building over time.

Some things you can consider to address the issue.
Easiest (simplest) to hardest (probably)

Make sure all your scheduled searches don't all run at once - ie, at 00 minutes every hour. Stagger them throughout the hour if you can. Avoid increments of 5 (00, 05,10 etc) and pick interval mins, (11,13,17) etc
Try to run heavy/long searches during otherwise quiet periods - overnight maybe? depends on your env.
Reduce the acceleration period - has more impact if you are building DMs from scratch, but shorter accel windows means less data - so shorter accel searches. (be mindful of the impact this has to your users! ES/ITSI!)
Add more Search heads (if SHclustered) but only applicable if your search peers are not also overwhelmed
Add more Cores to stand alone SHs - as above ^ only if your sure idx peers are not pegged.
If the peers are slammed...
Add more Search Peers
Add more Search Peer cores
Add more Search peer Iops

If you get far enough down that list to consider adding more compute/disk, it's probably worth checking that your environment is properly architected before adding resources to solve the problem. A detailed dive into where the bottleneck would be recommended.

Re: Data model acceleration not working

sarit_s — Sun, 23 Feb 2020 06:28:17 GMT

thank you very much for the detailed answer !
we are running with stand alone SH
we already decided that we want to add at least one more and clustered them but i didn't know that the issue with the data model is related

thanks,

Re: Data model acceleration not working

sarit_s — Sun, 23 Feb 2020 06:28:52 GMT

if you can, set your comment as an answer so i will approve it 🙂