Splunk Enterprise Security

Splunk ES datamodels broke (search delayed) after upgrade

icosine
Engager

Hi All,

We recently upgraded our Splunk Enterprise from V7.x to 8.x. After the upgrade, the security team observed that some searches are delayed, and mostly due to the data model acceleration from Splunk ES. 

Sample logs and screenshot below. We do not want to disable acceleration on default datamodels.. so how can we fix this issue? Note that there was no such issue before the upgrade..

 

03-05-2021 09:54:37.135 +0800 INFO SavedSplunker - savedsearch_id="nobody;Splunk_SA_CIM;_ACCELERATE_DM_Splunk_SA_CIM_Performance_ACCELERATE_", search_type="datamodel_acceleration", user="nobody", app="Splunk_SA_CIM", savedsearch_name="_ACCELERATE_DM_Splunk_SA_CIM_Performance_ACCELERATE_", priority=highest, status=success, digest_mode=1, scheduled_time=1614908940, window_time=0, dispatch_time=1614909165, run_time=110.969, result_count=431, alert_actions="", sid="scheduler__nobody_U3BsdW5rX1NBX0NJTQ__RMD5534aac642f80d961_at_1614908940_35488",

 

icosine_0-1614909427787.png

 

Labels (3)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Datamodel accelerations have the lowest priority so if they are delayed then it's because the system is busy processing other search types.

Check the start times of the alerts and scheduled searches to make sure they not bunched up.  Use the Monitoring Console to examine the search concurrency for high spots then adjust search schedules to lower them.

Making alerts and scheduled searches more efficient will help them run faster and reduce the delays in running lower-priority searches.

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Monitoring MariaDB and MySQL

In a previous post, we explored monitoring PostgreSQL and general best practices around which metrics to ...

Financial Services Industry Use Cases, ITSI Best Practices, and More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Splunk Federated Analytics for Amazon Security Lake

Thursday, November 21, 2024  |  11AM PT / 2PM ET Register Now Join our session to see the technical ...