topic Why is my search for missing events returning results from clients that actually do have events? in Splunk Search

Why is my search for missing events returning results from clients that actually do have events?

colbymahan — Wed, 07 Dec 2016 13:43:10 GMT

SourceName="EBS Check" OR SourceName="EBS Snapshot" | eval hasEBSCheck=1 | append [| metadata type="hosts" | eval hasEBSCheck=0] | stats max(hasEBSCheck) as hasEBSCheck by host | search hasEBSCheck=0 | search host=*-SLX NOT host=*-STG NOT host=*-SANDBOX NOT host=demo-slx

This search is returning 3 clients as having 0 events in the past day from those SourceNames, but they do in fact have the events. Only 3 of 250+ are coming back incorrectly. What could be reasons that these 3 are incorrectly calculated? Clicking on "Events" shows that in fact they have them and the count should not be "0".

Re: Why is my search for missing events returning results from clients that actually do have events?

somesoni2 — Wed, 07 Dec 2016 20:31:12 GMT

Give this a try

SourceName="EBS Check" OR SourceName="EBS Snapshot" | stats count as hasEBSCheck by host | append [| metadata type="hosts" | table host | eval hasEBSCheck=0] | stats max(hasEBSCheck) as hasEBSCheck by host | search hasEBSCheck=0 | search host=*-SLX NOT host=*-STG NOT host=*-SANDBOX NOT host=demo-slx

Re: Why is my search for missing events returning results from clients that actually do have events?

lguinn2 — Wed, 07 Dec 2016 21:09:27 GMT

I rewrote the search slightly; I think this makes it more efficient, but I don't believe that it makes the search more accurate.

SourceName="EBS Check" OR SourceName="EBS Snapshot" host=*-SLX NOT host=demo-slx 
| eval hasEBSCheck=1 
| stats first(hasEBSCheck) as hasEBSCheck by host
| append  [ metadata type="hosts" 
       | search host=*-SLX NOT host=demo-slx 
       | eval hasEBSCheck=0 ] 
| stats max(hasEBSCheck) as hasEBSCheck by host 
| search hasEBSCheck=0

To see if anything is going wrong with a search (either this one or yours), you can take a look at the Search Job Inspector and the search.log. Run the search, and then click the Job> link underneath the search bar to find the Job Inspector to examine how the search executed. The search job inspector also has a link to the search.log at the bottom of the page.

Re: Why is my search for missing events returning results from clients that actually do have events?

lguinn2 — Wed, 07 Dec 2016 21:10:30 GMT

I like the "stats count as hasEBSCheck"
Very efficient

Re: Why is my search for missing events returning results from clients that actually do have events?

colbymahan — Wed, 07 Dec 2016 22:03:16 GMT

While this "works" like the other search, it still is returning 3 hosts that show 0 in the chart but when you look, the events are there. So same results... Thanks for the suggestion though...

Re: Why is my search for missing events returning results from clients that actually do have events?

colbymahan — Tue, 13 Dec 2016 15:32:08 GMT

OK I think I know why now, but not how to fix it. Those hosts have events with differing case on the hostname from previous servers that were upgraded. It looks like splunk is seeing aic-slx as a different server than AIC-SLX. Since there are no recent events for the older name, it is returning that as a result. How can I get the report to see aic-slx and AIC-SLX as the same server? Adding NOT AIC-SLX causes the host to not bee included at all for both cases since Search is case insensitive. It seems like the lookup/meta data is the part that is differentiating case to see 2 different servers.