The first bug we noticed is triggered when using NOT in the where clause of tstats commands using datamodel summaries. To summarise it:
| tstats summariesonly=t count from datamodel=Authentication where NOT ( index=myindex ) by index sourcetype | tstats summariesonly=t count from datamodel=Authentication where NOT ( sourcetype=mysourcetype ) by index sourcetype | tstats summariesonly=t count from datamodel=Authentication where NOT ( index=myindex sourcetype=mysourcetype ) by index sourcetype
The first 2 searches returns results, as they should, but the last one returns nothing at all. Awful for accuracy!
The other bug affects raw searches with subsearches. Any kind of subsearch, even a foreach command. For instance:
index="_audit" sourcetype=audittrail action=success src!=127.0.0.1 login attempt NOT *uba | search [| stats count as index | eval index="*" ]
This search completely ignores the "action=success " and the "src!=127.0.0.1". This yields a lot more results than it should, also bad for accuracy.
I'd be curious if people on different versions of splunk (especially 7.3.*) can reproduce these.
If you're thinking to upgrade, make sure you test for it before. As far as I can see, I do not believe that these issues are mentioned in the known issues in the release notes, even though splunk is aware that something is wrong.
We have open cases with splunk support and unfortunately, at least for the first bug, there doesn't seem to be a fix coming any time soon.
Your searches are not following best practices for clarity/hygiene.
Try adding an explicit
AND like this:
| tstats summariesonly=t count from datamodel=Authentication where NOT ( index=myindex AND sourcetype=mysourcetype ) BY index sourcetype
Try getting rid of the
| search and using correct
subsearch format like this:
index="_audit" AND sourcetype=audittrail AND action=success AND src!=127.0.0.1 AND login AND attempt AND NOT *uba AND [| makeresults | eval index="*" ]
P.S. Your subsearch as written does nothing.
I did try your suggestions to leave no stones unturned, but as I expected it made no difference. I do know the subsearch does nothing, it's just to illustrate that any subsearch triggers the bug, before or after the first pipe. Even the following triggers the bug:
index="_audit" sourcetype=audittrail action=success src!=127.0.0.1 login attempt NOT *uba | foreach blah* [ eval foo_<<MATCHSTR>>=lower(<<FIELD>>) ]"
Version 8 made many changes to optimize searches. Have you tried adding
| noop search_optimization=false to ends of your queries?