Why is my search for missing events returning results from clients that actually do have events?

[colbymahan]

SourceName="EBS Check" OR SourceName="EBS Snapshot" | eval hasEBSCheck=1 | append [| metadata type="hosts" | eval hasEBSCheck=0] | stats max(hasEBSCheck) as hasEBSCheck by host | search hasEBSCheck=0 | search host=*-SLX NOT host=*-STG NOT host=*-SANDBOX NOT host=demo-slx 

This search is returning 3 clients as having 0 events in the past day from those SourceNames, but they do in fact have the events. Only 3 of 250+ are coming back incorrectly. What could be reasons that these 3 are incorrectly calculated? Clicking on "Events" shows that in fact they have them and the count should not be "0".

[colbymahan]

OK I think I know why now, but not how to fix it. Those hosts have events with differing case on the hostname from previous servers that were upgraded. It looks like splunk is seeing aic-slx as a different server than AIC-SLX. Since there are no recent events for the older name, it is returning that as a result. How can I get the report to see aic-slx and AIC-SLX as the same server? Adding NOT AIC-SLX causes the host to not bee included at all for both cases since Search is case insensitive. It seems like the lookup/meta data is the part that is differentiating case to see 2 different servers.

[lguinn2]

I rewrote the search slightly; I think this makes it more efficient, but I don't believe that it makes the search more accurate.

SourceName="EBS Check" OR SourceName="EBS Snapshot" host=*-SLX NOT host=demo-slx 
| eval hasEBSCheck=1 
| stats first(hasEBSCheck) as hasEBSCheck by host
| append  [ metadata type="hosts" 
       | search host=*-SLX NOT host=demo-slx 
       | eval hasEBSCheck=0 ] 
| stats max(hasEBSCheck) as hasEBSCheck by host 
| search hasEBSCheck=0 

To see if anything is going wrong with a search (either this one or yours), you can take a look at the Search Job Inspector and the search.log. Run the search, and then click the Job> link underneath the search bar to find the Job Inspector to examine how the search executed. The search job inspector also has a link to the search.log at the bottom of the page.

[somesoni2]

Give this a try

SourceName="EBS Check" OR SourceName="EBS Snapshot" | stats count as hasEBSCheck by host | append [| metadata type="hosts" | table host | eval hasEBSCheck=0] | stats max(hasEBSCheck) as hasEBSCheck by host | search hasEBSCheck=0 | search host=*-SLX NOT host=*-STG NOT host=*-SANDBOX NOT host=demo-slx

[colbymahan]

While this "works" like the other search, it still is returning 3 hosts that show 0 in the chart but when you look, the events are there. So same results... Thanks for the suggestion though...

[lguinn2]

I like the "stats count as hasEBSCheck"
Very efficient