Installation

Major bugs in searches in splunk 7.3 DO NOT UPGRADE

gabriel_vasseur
Contributor

The first bug we noticed is triggered when using NOT in the where clause of tstats commands using datamodel summaries. To summarise it:

| tstats summariesonly=t count from datamodel=Authentication where NOT ( index=myindex ) by index sourcetype
| tstats summariesonly=t count from datamodel=Authentication where NOT ( sourcetype=mysourcetype ) by index sourcetype
| tstats summariesonly=t count from datamodel=Authentication where NOT ( index=myindex sourcetype=mysourcetype ) by index sourcetype

The first 2 searches returns results, as they should, but the last one returns nothing at all. Awful for accuracy!

The other bug affects raw searches with subsearches. Any kind of subsearch, even a foreach command. For instance:

index="_audit" sourcetype=audittrail action=success src!=127.0.0.1 login attempt NOT *uba
| search [| stats count as index | eval index="*" ]

This search completely ignores the "action=success " and the "src!=127.0.0.1". This yields a lot more results than it should, also bad for accuracy.

I'd be curious if people on different versions of splunk (especially 7.3.*) can reproduce these.

If you're thinking to upgrade, make sure you test for it before. As far as I can see, I do not believe that these issues are mentioned in the known issues in the release notes, even though splunk is aware that something is wrong.

We have open cases with splunk support and unfortunately, at least for the first bug, there doesn't seem to be a fix coming any time soon.

Labels (1)

mhoogcarspel_sp
Splunk Employee
Splunk Employee

The subsearch one was reported as SPL-176990 is fixed in 7.3.3
for where tstats where, reported as SPL-179746, fix planned for 7.3.4 at the moment

woodcock
Esteemed Legend

Your searches are not following best practices for clarity/hygiene.

Try adding an explicit AND like this:

| tstats summariesonly=t count from datamodel=Authentication where NOT ( index=myindex AND sourcetype=mysourcetype ) BY index sourcetype

Try getting rid of the | search and using correct subsearch format like this:

index="_audit" AND sourcetype=audittrail AND action=success AND src!=127.0.0.1 AND login AND attempt AND NOT *uba AND [| makeresults | eval index="*" ]

P.S. Your subsearch as written does nothing.

0 Karma

gabriel_vasseur
Contributor

I did try your suggestions to leave no stones unturned, but as I expected it made no difference. I do know the subsearch does nothing, it's just to illustrate that any subsearch triggers the bug, before or after the first pipe. Even the following triggers the bug:

index="_audit" sourcetype=audittrail action=success src!=127.0.0.1 login attempt NOT *uba
| foreach blah* [ eval foo_<<MATCHSTR>>=lower(<<FIELD>>) ]"

richgalloway
SplunkTrust
SplunkTrust

Version 8 made many changes to optimize searches. Have you tried adding | noop search_optimization=false to ends of your queries?

---
If this reply helps you, Karma would be appreciated.

gabriel_vasseur
Contributor

We haven't tried version 8 yet. Adding | noop search_optimization=false has no impact on either bug.

0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

Industry Solutions for Supply Chain and OT, Amazon Use Cases, Plus More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Enterprise Security Content Update (ESCU) | New Releases

In November, the Splunk Threat Research Team had one release of new security content via the Enterprise ...