Splunk Enterprise Security

Splunk ES datamodels broke (search delayed) after upgrade

icosine
Engager

Hi All,

We recently upgraded our Splunk Enterprise from V7.x to 8.x. After the upgrade, the security team observed that some searches are delayed, and mostly due to the data model acceleration from Splunk ES. 

Sample logs and screenshot below. We do not want to disable acceleration on default datamodels.. so how can we fix this issue? Note that there was no such issue before the upgrade..

 

03-05-2021 09:54:37.135 +0800 INFO SavedSplunker - savedsearch_id="nobody;Splunk_SA_CIM;_ACCELERATE_DM_Splunk_SA_CIM_Performance_ACCELERATE_", search_type="datamodel_acceleration", user="nobody", app="Splunk_SA_CIM", savedsearch_name="_ACCELERATE_DM_Splunk_SA_CIM_Performance_ACCELERATE_", priority=highest, status=success, digest_mode=1, scheduled_time=1614908940, window_time=0, dispatch_time=1614909165, run_time=110.969, result_count=431, alert_actions="", sid="scheduler__nobody_U3BsdW5rX1NBX0NJTQ__RMD5534aac642f80d961_at_1614908940_35488",

 

icosine_0-1614909427787.png

 

Labels (3)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Datamodel accelerations have the lowest priority so if they are delayed then it's because the system is busy processing other search types.

Check the start times of the alerts and scheduled searches to make sure they not bunched up.  Use the Monitoring Console to examine the search concurrency for high spots then adjust search schedules to lower them.

Making alerts and scheduled searches more efficient will help them run faster and reduce the delays in running lower-priority searches.

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!