Splunk ES datamodels broke (search delayed) after upgrade

[icosine]

Hi All,

We recently upgraded our Splunk Enterprise from V7.x to 8.x. After the upgrade, the security team observed that some searches are delayed, and mostly due to the data model acceleration from Splunk ES. 

Sample logs and screenshot below. We do not want to disable acceleration on default datamodels.. so how can we fix this issue? Note that there was no such issue before the upgrade..

03-05-2021 09:54:37.135 +0800 INFO SavedSplunker - savedsearch_id="nobody;Splunk_SA_CIM;_ACCELERATE_DM_Splunk_SA_CIM_Performance_ACCELERATE_", search_type="datamodel_acceleration", user="nobody", app="Splunk_SA_CIM", savedsearch_name="_ACCELERATE_DM_Splunk_SA_CIM_Performance_ACCELERATE_", priority=highest, status=success, digest_mode=1, scheduled_time=1614908940, window_time=0, dispatch_time=1614909165, run_time=110.969, result_count=431, alert_actions="", sid="scheduler__nobody_U3BsdW5rX1NBX0NJTQ__RMD5534aac642f80d961_at_1614908940_35488",

[richgalloway]

Datamodel accelerations have the lowest priority so if they are delayed then it's because the system is busy processing other search types.

Check the start times of the alerts and scheduled searches to make sure they not bunched up.  Use the Monitoring Console to examine the search concurrency for high spots then adjust search schedules to lower them.

Making alerts and scheduled searches more efficient will help them run faster and reduce the delays in running lower-priority searches.