Splunk Search

Search for tag!=foo when "foo" doesn't exist as a tag returns nothing instead of all events.

splunk_a_tron
Engager

Hello All,

I am new to Splunk and ran into my first wall when attempting to omit search results using tags. Any help on the following would be greatly appreciated.

Use Case: We want to be able to hide/omit/delete certain events when needed. (i.e. there is an infrastructure issue that results in a days worth of events being erroneous and we do not want to include this data for historical reporting, alerting, etc. as it will throw off certain calcs like the 90 day average)

Constraints: Our Splunk admin said that they will not "delete" events except under certain circumstances (so we cannot use "|delete" as an option)

Tentative Solution: We plan to adopt the use of a tag "snooze" that we will use to suppress such "bad data" in all our searches/reports/alerts/etc.

Issue: The following works IF the tag "snooze" exists on at least one event ('Splunk search'  tag!=snooze) - this correctly returns all events that do not have a "snooze" tag on any fields. However, if NO events exist with the "snooze" tag then the search returns no results (when it should instead be returning ALL events).

Is there a way to change the evaluation so that all events are returned when a given tag doesn't currently exist? The likely scenario is that we apply the tag on a given record and that record may age out which would cause all our reports to break.

Additionally, if there is a better approach to what we are trying to achieve I am all ears.

Thank you,

Splunk-a-tron

Labels (2)
0 Karma
1 Solution

Richfez
SplunkTrust
SplunkTrust

Well, that means that many of the ones with tag=snooze also have another tag set on it.  Which should be perfectly fine, but ... hmm.  Deserves looking into.  Though there are 10 "snoozes" in that list.  That means "NOT tag=snooze" should only hit 2 of them anyway.

There's few enough items we could just manually check them.

index="nameofourindex" test
| table tag

Oooh another thought too.

index="nameofourindex" test tag=* tag!=snooze

vs.

index="nameofourindex" test tag=* NOT tag=snooze

That's testing only where tag is set, let's see if the two numbers from those add up to 10.

And honestly, maybe it's time to open a support ticket?  I'm sure if you can get someone to listen, and actually listen close and understand this isn't a question about NOT vs. != (well, it is, but in an anomalous sense, not in the usual sense), that they might be able to tell you something.

View solution in original post

0 Karma

splunk_a_tron
Engager

Hi Rich,

Thanks for such a quick response! I actually tried both previously and get the same results with each:

1) 'NOT tag="sooze"'      -  This returns the correct results if at least one event has a tag of "snooze", but returns no results if no events have a tag of "snooze"

2) 'tag!="sooze"'      -  This returns the correct results if at least one event has a tag of "snooze", but returns no results if no events have a tag of "snooze"

I agree that based on what I researched that I thought #1 above would have worked, but I got the same result with both so I reverted back to #2 in my original question.

Any other thoughts as to why #1 may not be working?

0 Karma

Richfez
SplunkTrust
SplunkTrust

Really!  huh.

Can you provide an example of each category of event?

Because no, except for the typo I probably made "sooze" vs. "snooze" I can't think of any reason this wouldn't work so I'm a little puzzled.  Feels like if it were an actual bug, it would affect nearly everyone and be well known, and since that's not the case I suspect we're just missing something simple here.

Which means *sigh* we start with the basics of getting a few lines of sample data.  You can clean them up and obfuscate bits if you need to, and I don't think we need all fields, just a representative sample of them with the important fields there (e.g. "tag" lol). 

Also the actual search you are using, if possible?

0 Karma

splunk_a_tron
Engager

Hi Rich,

This is where my newness will likely shine, but when you say "category of event" do you just mean examples of the different types of events in the index that I am searching? If so, the following are some examples:

Currently we have an index where we are sending custom formatted events via a python script based on calculated data form Corvil logs. We also sent in some test events, all of which include the word "test" somewhere in the event. The base search I am using to try and test the tags is simply 'index="nameofourindex" test. This returns approx. a dozen events that look like either of the following (note: event #1 has two fields that are pipe separated, whereas event#2 just has a string of "test log" in the raw text):

Event Example 1: Time (date/Time) :  Event (Text=Monitoring Log Test| Type=LOG4)

Event Example 2: Time (date/Time) :  test log

The rest of the events in the index are the Corvil data events that have Time (date/Time) and then approx.  16 or so pipe separated fields.

I selected events within the 12 that contain the word "test" and added the "snooze" label to those with "Type=LOG4", "Type=LOG5", etc. so that all the "test" events have a "snooze" label with the exception of those two that just have "test log" as raw text, which have no labels.

If I add NOT tag=snooze then I only get the two events that have "test" but not a "snooze" label (so that works)

If I change this to NOT tag=foo (where foo is not a currently used tag) I get zero results (when I would think I should get all 12 events that have "test"

Apologies if my explanation is confusing or I am not providing the info you need, I appreciate your help and patience.

0 Karma

Richfez
SplunkTrust
SplunkTrust

Oh I wonder if they're not being parsed out properly as fields, so it's really doing string comparisons -

So `NOT tag=X` is actually looking to do a `NOT <string "tag=X">` instead of `NOT <field tag>=<value X>`.

I'm trying to wrap my brain around that, if it's possible, and if it's possible is that the problem here?

OK, let's leave that aside for just a second, I'll get back to it.

So here's how it should work - probably all stuff you've figured out, but feels useful to just get it out here for others.  If you run this search (all by itself), you'll see I mock up some data, LOG1-LOG5 events.  Then I conditionally create a tag field, for LOG2 and LOG4, it's set to "snooze".    For grins and giggles, I also set LOG3's tag to "otherTag" just to illustrate all the possibilities.

| makeresults 
| eval Type="LOG1 LOG2 LOG3 LOG4 LOG5"
| makemv Type | mvexpand Type
| eval tag = if(Type=="LOG2" OR Type=="LOG4", "snooze", null())
| eval tag = if(Type=="LOG3", "otherTag", tag)

Now, to the end of that if you append

| search NOT tag=snooze

It will return three rows; the two (LOG1, LOG5) where tag isn't defined at all, and LOG3 where tag is defined but it's not "snooze"

Alternatively, if you change that last trailing `| search ...` to be

| search tag!=snooze

You'll get back one row for LOG3, because it's the only row that has tag defined, and yet where tag is not "snooze".

So, to come full circle, here's what I think is happening.

Your "field" isn't a field, and this needs to be fixed.  You don't have a field named "tag" with a value or or not of "snooze", you instead have a string that has an equals sign in it. "tag=snooze".

The way to tell is, if it's a real field, run your search in Verbose mode (upper right, by the time picker - Fast, Smart and Verbose dropdown), and then on the left hand side check the fields - if you don't have one called "tags" which when you click on it it gives you values like 'snooze' and whatever, then it's not a "field" yet and is just a string.

Automatic field extraction should have taken care of this, and since it didn't I suspect you have a data onboarding issue.  Perhaps you need `KV_MODE = json` on the stanza in props.conf for that sourcetype?  You can see that information here: https://docs.splunk.com/Documentation/Splunk/8.1.0/Knowledge/Automatickey-valuefieldextractionsatsea...

There's ways to fix this after the fact (actually, KV_MODE = json fixes it after the fact too because it's search time), but maybe you can run ` | spath blah blah` on it, or perhaps in this one case we can eval it into a new field or rex it.  But either way, investigate why it's not a field and get that fixed, and I think everything else will just work.

 

Let us know what you find,

Rich

0 Karma

splunk_a_tron
Engager

Thanks Rich, I checked and the tags do indeed appear to be fields as they show up on the left-hand side with all the other fields. I also tried to see if it was how I was adding the tags and I tried via selecting a given field in search and adding the label as well as creating an event type and assign those events a label but the behavior stays the same. I might try this on other index and see if I still have the same issue.

0 Karma

Richfez
SplunkTrust
SplunkTrust

I am puzzled then.

Back to basics, then, let's see what thing we're missing.

index="nameofourindex" test

Returns 12 events?

index="nameofourindex" test tag="snooze"

Returns some smaller number of events, let's say N.

index="nameofourindex" test NOT tag="snooze"

Should therefore return 12-N=M events, so if N is 4 from the above, this would be 8.

 

Is that or is that not the case?  If your search is a little more complex, please post it here in case there's some nuance with parenthesis or something we need to investigate.

Don't worry, we'll get this.  Just might take a bit of slow going for a bit to make sure we're not overlooking that one thing... 🙂

-Rich

0 Karma

splunk_a_tron
Engager

Yup, you got it Rich. This is all correct and the search really is just the index and "test". Then if you change the last example to:

index="nameofourindex" test NOT tag="tagthatdoesntexistinsplunk"

 
the search returns "No Results" (instead of what I/we believe should be 12).

I also just tested this with event types that I created, to see if I could use that instead, and the same behavior occurs. If I attempt to search for all events that are not a given eventype (but that event type doesn't exist in splunk) I also get "no results".

0 Karma

Richfez
SplunkTrust
SplunkTrust

All right, two things - one super minor bookkeeping thing that I was doing to, another that is actually pertinent.

Minor bookkeeping thing - I keep clicking the wrong "reply" button so the threading is breaking. Either that or the new platforms threading is still bad, not sure.  Anyway, I am a moderator so I am going to start trying to shuffle things around to thread this all right.

The real thing -

Let's check the generated LISPY for each thing.

For a search that works, and one that doesn't, run the search.  Then click the job dropdown near the top of the data/output pane.  Select "Inspect job" and it'll open a new window. 

In there, click the link to open the search.log (it's either a link right near the top, or all the way at the bottom).

Search in there for "base lispy" and when you find it, copy that out and paste it back into an answer here.

It may be useful to search for ERROR and see if anything matches that and include it too.

 

Then repeat with a search that doesn't behave properly.

I'm not yet sure what this will tell us, but maybe it'll be useful.  It often is.

-Rich

 

0 Karma

Richfez
SplunkTrust
SplunkTrust

(Also never mind I'm not moving messages, that function does not do what I thought it does, and looks pretty scary.  🙂 )

0 Karma

Richfez
SplunkTrust
SplunkTrust

Also maybe just try

index="nameofourindex" test tag="snooze"
| stats count by tag

Possibly we'll see something useful in that.

 

0 Karma

splunk_a_tron
Engager

When I run that it shows me that I have a total of 4 tags (which was to be expected), with 10 of 12 events having a tag of snooze. Should it have also enumerated the counts of the other 3 tags (because it did). so:

tagcount
error8
log8
snooze10
test10

 

0 Karma

Richfez
SplunkTrust
SplunkTrust

Well, that means that many of the ones with tag=snooze also have another tag set on it.  Which should be perfectly fine, but ... hmm.  Deserves looking into.  Though there are 10 "snoozes" in that list.  That means "NOT tag=snooze" should only hit 2 of them anyway.

There's few enough items we could just manually check them.

index="nameofourindex" test
| table tag

Oooh another thought too.

index="nameofourindex" test tag=* tag!=snooze

vs.

index="nameofourindex" test tag=* NOT tag=snooze

That's testing only where tag is set, let's see if the two numbers from those add up to 10.

And honestly, maybe it's time to open a support ticket?  I'm sure if you can get someone to listen, and actually listen close and understand this isn't a question about NOT vs. != (well, it is, but in an anomalous sense, not in the usual sense), that they might be able to tell you something.

0 Karma

splunk_a_tron
Engager

Ok, I think we have it figured out now . . .

It looks like the issue is that both search attempts are predicated on the existence of the tag you "don't want". The LISPY/Search Inspect shows the following for these:

Search:

 

index="nameofourindex" test NOT tag="tagthatdoesntexistinsplunk"

 

 
Result: 

 

info: The specified search will not match any events
warn: Unable to find tag "tagthatdoesntexistinsplunk"

 

 
Search:

 

index="nameofourindex" test tag!="tagthatdoesntexistinsplunk"​

 


Result: 

 

info: The specified search will not match any events
warn: Unable to find tag "tagthatdoesntexistinsplunk"

 

 


Your other suggestions work, but I believe this would require that all events have at least one tag:

 

index="nameofourindex" test tag=* NOT tag=snooze
OR
index="nameofourindex" test tag=* tag!=snooze

 

 
Finally, I tried the following and this seems to do the trick. (However, does this mean the "NOT tag=tagthatdoesntexistinsplunk" isn't working as it should in that it should also include events with no tag at all in the search results?)

 

index="nameofourindex" test (tag=* NOT tag="tagthatdoesntexistinsplunk") OR (NOT tag=*)

 

 
Thanks again for all the help Rich, I was hoping this would be a layup question but it proved a little more tricky than I thought. I appreciate your time!

Splunk_a_tron

0 Karma

Richfez
SplunkTrust
SplunkTrust

I think if I am reading this right, you are using `tag!="snooze"` when you actually want `NOT tag="snooze"`.

The former, `tag!="snooze"` returns all events where there is a tag defined and exists, and where that tag is not "snooze."

The latter, `NOT tag="snooze"`, returns all events that do not have a tag that isn't equal to "snooze", or in fact that may not have a tag defined at all.  Doesn't matter - no tag defined means it's certainly not defined as "snooze" right?

So use `NOT tag="snooze"` and see if that works better for you.

https://docs.splunk.com/Documentation/Splunk/8.1.0/Search/NOTexpressions

Happy Splunking!

-Rich

 

 

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...