Search '!=' does not work after upgrading to Splunk 8

[girtsgr]

Anybody else having issues with search operator '!=' after upgrading to Splunk Enterprise 8?

My search is index=myindex | search src_ip!=10.0.0.0/8 and after the upgrade Splunk started to return all the local addresses in the search results. Any ideas what to do?

[mbrunetto]

Worked a similar issue with support, but couldn't use the "where" option due to the way my search was formed. You can also make a change to limits.conf to use the old behavior. Documenting here in case anyone else finds this article looking for a similar solution.

Workaround:
limits.conf: [search] use_search_evaluator_v2=false

Tracked as Splunk Issues: SPL-179357, SPL-179700
Reference:
https://docs.splunk.com/Documentation/Splunk/8.0.0/ReleaseNotes/Knownissues

[garias_splunk]

There is a workaround, try this:
use a "| where" command instead, example:
index=_internal
| where NOT cidrmatch("127.0.0.0/8", clientip)
| stats count BY clientip

[girtsgr]

Yeah, already got the support email and implemented it in some cases, and also posted here.

[mhoogcarspel_sp]

Just to be clear, it's the combination of:
Splunk 8.0.0
search
negation (!=, NOT)
in combination with CIDR based filtering (10.0.0.0/8)

where command with cidrmatch isn't affected
negation of non-CIDR values isn't affected

[girtsgr]

Fails without search command as well, eg, index=myindex src_ip!=10.0.0.0/8

But yes:

where command with cidrmatch isn't affected
negation of non-CIDR values isn't affected
true

[mhoogcarspel_sp]

"index=myindex src_ip!=10.0.0.0/8" is also the search command, any search string without a command at the beginning is actually "| search ... " 🙂

[martin_mueller · Accepted solution]

Looks like a bug to me, I've filed a case ¯\(ツ)/¯

[martin_mueller]

Repro'd by support, bug SPL-179357.

[girtsgr]

FYI: support suggested a workaround:

There is a workaround, try this:
use a "| where" command instead, example:
index=_internal
| where NOT cidrmatch("127.0.0.0/8", clientip)
| stats count BY clientip

[niketn]

Adding a where to second pipe would mean query will become expensive. You will be pulling all IPs from index then filtering out one is a very bad search to execute. This can be a workaround only if your Splunk Infrastructure can support and only if dashboard somehow still performs.

[martin_mueller]

Not really, no. The original NOT field=value for search-time fields is slow already, not making it to lispy. Moving it to a where won't change scanCount.

[niketn]

I did NOT see that. Thanks for correction 🙂

[girtsgr]

Is it worth to file another report? Will that help to escalate the priority of this bug?

[martin_mueller]

Can't hurt to tell support that you're affected too. At the very least they'll then tell you when a fix is available.

[girtsgr]

Thanks, done.

[manjunathmeti]

Try index=myindex src_ip!="10.0.0.0/8" OR index=myindex NOT src_ip="10.0.0.0/8".

[girtsgr]

Tried it already, no luck.