Reporting

Splunk Searches delayed after upgrading ES to 6.4.1

jerm1020rq
Explorer

We recently upgraded a splunk environment to version 8.0.2 from a 7.2 version. Along with this, we upgraded Splunk Enterprise Security to version 6.4.1. Once the upgrade was completed, splunk shows a critical error related to delayed searches that was not an issue before. I did notice that there are quite a few data models accelerated and suspect this could be the cause. Has anyone had an issue with Splunk having delayed searches post upgrade or possibly data models being automatically accelerated post upgrade? This was not an issue prior so I feel it is most likely related to the ES or Splunk upgrade itself. Screen Shot 2021-06-01 at 3.13.15 PM.png

Labels (4)
0 Karma

jschroederevers
Explorer

We ran into this same issue when upgrading to 8.1.2 (and ES to 6.4.1). It looks to be related to changes in the Health Report Manager, as there are some new feature indicators. You can disable or modify the thresholds in the UI or in health.conf (see https://docs.splunk.com/Documentation/Splunk/8.1.2/DMC/Configurefeaturemonitoring).

If you look at the internal logs, you can see if delayed searches have actually increased:

index=_internal component=savedsplunker reason=*
| timechart count

 As long as no other changes were introduced, I would expect the delayed search count to stay relatively stable. Our next step is to dig in to these searches to see if we can optimize.

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!