Splunk Enterprise Security

Splunk ES datamodels broke (search delayed) after upgrade

icosine
Engager

Hi All,

We recently upgraded our Splunk Enterprise from V7.x to 8.x. After the upgrade, the security team observed that some searches are delayed, and mostly due to the data model acceleration from Splunk ES. 

Sample logs and screenshot below. We do not want to disable acceleration on default datamodels.. so how can we fix this issue? Note that there was no such issue before the upgrade..

 

03-05-2021 09:54:37.135 +0800 INFO SavedSplunker - savedsearch_id="nobody;Splunk_SA_CIM;_ACCELERATE_DM_Splunk_SA_CIM_Performance_ACCELERATE_", search_type="datamodel_acceleration", user="nobody", app="Splunk_SA_CIM", savedsearch_name="_ACCELERATE_DM_Splunk_SA_CIM_Performance_ACCELERATE_", priority=highest, status=success, digest_mode=1, scheduled_time=1614908940, window_time=0, dispatch_time=1614909165, run_time=110.969, result_count=431, alert_actions="", sid="scheduler__nobody_U3BsdW5rX1NBX0NJTQ__RMD5534aac642f80d961_at_1614908940_35488",

 

icosine_0-1614909427787.png

 

Labels (3)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Datamodel accelerations have the lowest priority so if they are delayed then it's because the system is busy processing other search types.

Check the start times of the alerts and scheduled searches to make sure they not bunched up.  Use the Monitoring Console to examine the search concurrency for high spots then adjust search schedules to lower them.

Making alerts and scheduled searches more efficient will help them run faster and reduce the delays in running lower-priority searches.

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...