The problem
Sometimes specific events are missing from an accelerated data model. These events can be found with a regular SPL search.
When searching the data model, the events are only returned when the data model is not accelerated. Once the acceleration is enabled, the events do not show up any more. Of course, we checked that the DMA's status is "100% completed". Rebuilding the accelerated data model does not help.
What is really strange: it happens that events are disappearing from the accelerated data model that showed up just fine mere minutes earlier. I noticed that today when I refreshed a dashboard and some charts suddenly were empty (which led me to investigate this again and post this question).
An example search
Here is an example of a search that is not returning all expected events (we have noticed this issue with different datasets in the past, too):
| pivot uberAgent Process_NetworkTargetPerformance
count(Process_NetworkTargetPerformance) as "Event count"
splitrow AppName
filter host is "Client11"
We have seen this issue on various versions of Splunk Enterprise 7.1.x and 7.2.x (including 7.2.1).
Screenshots
Screenshots of the above search, first with acceleration disabled (9 results), then with acceleration enabled (6 results):
Scheduler.log
I ran a search to query scheduler.log over a time range of 8 hours, which covered the time of the issue:
index=_internal source=*scheduler.log
| stats count by log_level
Result:
DEBUG: 22537
ERROR: 3
INFO: 5811
Update 2020-02-04: back in Splunk 8.x
Dominik's answer fixed the problem for Splunk versions 7.x. However, it is back in Splunk 8 (we tested 8.0 and 8.0.1).
... View more