topic Re: Negative Event searches in Splunk Search

Negative Event searches

jsmith39 — Thu, 30 Jan 2014 19:11:55 GMT

How would you search an application log for the absence of one or more specific events in a given time period?

I'm looking through event logs on multiple servers for specific records and am generating a report from them. So if an event of 12345 transpires, it's diplayed by this
sourcetype=applicationlogs tag=serverset eventcode=12345 | stats count by host

if it doesn't then that particular server just isn't listed in the report output. I'd like to note the absence and state "didn't happen".

Re: Negative Event searches

theeven — Thu, 30 Jan 2014 19:30:20 GMT

If I understand correctly, here's what you looking for.

sourcetype=applicationlogs tag=serverset | 
eval status=if(match(eventcode, 12345), "yes", "no") | 
search status=no |
stats count by host

Re: Negative Event searches

martin_mueller — Thu, 30 Jan 2014 20:41:10 GMT

Here's a thought (untested):

| metadata type=hosts index=yourindex | fields host | join type=left host [search index=yourindex blah blah eventcode=12345 | stats count by host] | fillnull count

Rows with count=0 are the droids you're looking for.

Note, I'm assuming all hosts in that index are potential senders of eventcode=12345. If that's not the case then you need to compute a list of all relevant hosts there instead of using metadata.

Re: Negative Event searches

jsmith39 — Fri, 31 Jan 2014 16:07:11 GMT

theeven: I had to modify what you wrote to get it to run
tag=Backup| eval status=if(matcheventcode==34112, "yes", "no") | search status=no | stats count by host

Perhaps I modified it too much, but the end result is that I'm getting the book thrown at me as every record that isn't the matched eventcode is returned.

martin_mueller: Same scenario, I had to modify your query in order to run it, and perhaps I went too far but the results were not as expected. Strangely enough, there wasn't any statistics view and the records found were filtered, but I'm still trying to figure out in what way. The results are less than everything except 34112 but more than just noting an absence, for example I'm getting eventcode=0 and eventcode=58061 among other results.

tag=Backup | fields host | join type=left host [search tag=Backup eventcode=34112 | stats count by host] | fillnull count

Appreciate your help regardless

Re: Negative Event searches

somesoni2 — Fri, 31 Jan 2014 16:59:15 GMT

Try this:

sourcetype=applicationlogs tag=serverset 
| eval eventcode=if(match(eventcode, 34112), eventcode, "----") 
| stats count,values(eventcode) as eventcode  by host 
| eval isPresent=mvfind(eventcodes,"34112")
| eval count=if(isnull(isPresent),"Didn't happen", count)
| fields - isPresent, eventcode

Re: Negative Event searches

martin_mueller — Fri, 31 Jan 2014 18:18:33 GMT

Replacing the call to metadata with a search for some events does indeed change my search significantly, so that cannot work.You could replace metadata with this:

search for all events | dedup host | fields host

What didn't work for you?

Re: Negative Event searches

jsmith39 — Mon, 03 Feb 2014 18:05:55 GMT

Thanks for the input everyone, I ended up going a more brute force method that seems to be working.
I tagged all of the servers to be searched, then filtered them by SourceName so I only had a working pool of logs that were related to the software application I'm looking at. It's a bit more verbose than I wanted but I'm hoping as I get more data I'll be able to refine it.

tag=Backup SourceName="Backup Exec" | eval Outcome=case(EventCode==57755 OR EventCode==34112, "Success", EventCode==34113 OR EventCode==34114, "Failure", NOT EventCode==57755 OR EventCode==34112 OR EventCode==34113 OR EventCode==34114, "Missed Jobs") | stats count by host, Outcome | rename host as "Server Name"