Splunk Searches delayed after upgrading ES to 6.4.1

[jerm1020rq]

We recently upgraded a splunk environment to version 8.0.2 from a 7.2 version. Along with this, we upgraded Splunk Enterprise Security to version 6.4.1. Once the upgrade was completed, splunk shows a critical error related to delayed searches that was not an issue before. I did notice that there are quite a few data models accelerated and suspect this could be the cause. Has anyone had an issue with Splunk having delayed searches post upgrade or possibly data models being automatically accelerated post upgrade? This was not an issue prior so I feel it is most likely related to the ES or Splunk upgrade itself.

[jschroederevers]

We ran into this same issue when upgrading to 8.1.2 (and ES to 6.4.1). It looks to be related to changes in the Health Report Manager, as there are some new feature indicators. You can disable or modify the thresholds in the UI or in health.conf (see https://docs.splunk.com/Documentation/Splunk/8.1.2/DMC/Configurefeaturemonitoring).

If you look at the internal logs, you can see if delayed searches have actually increased:

index=_internal component=savedsplunker reason=*
| timechart count

 As long as no other changes were introduced, I would expect the delayed search count to stay relatively stable. Our next step is to dig in to these searches to see if we can optimize.