topic Re: Why is my alert triggering false positives? in Alerting

Why is my alert triggering false positives?

nmayafit — Wed, 19 Apr 2017 20:24:53 GMT

Hi,

I have a very strange issue and I'm trying to solve it in the last week with no luck.
I have an alert created from a percentage error rate of my server.
The search is simple:

host=*-prod* source="/var/log/xxx"  | stats count as All, count(eval(level="ERROR")) as ERROR | eval Alert=((ERROR/All)*100) | table Alert | where Alert >= 0.3

But, the alert is triggering false positive all the time.
Now, what strange is that when I push the "View Results" from email I see the false result (e.g 0.4), but when I just search the same search (just hit enter in the search field right after) I then get the right result (e.g 0.1).

These are my alert configurations:

Is this a bug? Am I doing something wrong?

Re: Why is my alert triggering false positives?

somesoni2 — Wed, 19 Apr 2017 20:50:17 GMT

Your alerts would be looking at a particular time range. Try to run the search manually for exactly same time range with hardcoded dates. Like if your alert ran at 4/19/2017 1:00 PM looking last 60 minutes, run the search for fixed time range of 4/19/2017 12:00 PM to 4/19/2017 1:00 PM and compare result.

Re: Why is my alert triggering false positives?

nmayafit — Wed, 19 Apr 2017 21:11:33 GMT

The alert and the manual search both look at the last 2 hours.
As I said, I take the exact search that I get from the "View Results" and run it again. Same query, same time range. Different results.

Re: Why is my alert triggering false positives?

somesoni2 — Tue, 29 Sep 2020 13:45:22 GMT

You get a summary line just below the search text box in the format "nnnnn events (xxxxstart_datexxx to yyyyend_dateyyyy)" . Check if they both are exactly same. Since you're using relative time, it may change (last 2hrs now will be different if you run it after 5 min from now).
If that is same, there may be some more events being ingested (success events) between the alert schedule and you manually running causing Error percent to go down. I would first eliminate timerange mismatch and they troubleshoot further.

Re: Why is my alert triggering false positives?

nmayafit — Thu, 20 Apr 2017 03:48:27 GMT

When I get from the "view results" it says only 1 result. When I do the same search again it gives me millions.
How can it be?

Re: Why is my alert triggering false positives?

somesoni2 — Thu, 20 Apr 2017 04:55:39 GMT

The clicking of "view result" result load the result from dispatch directory which it finds 1 events (output of stats). The re-run of the search scans all relative events from base search hence that higher count. Do the time range matches when you click the "view results" versus running search again?

Re: Why is my alert triggering false positives?

nmayafit — Thu, 20 Apr 2017 05:30:54 GMT

I see the same time range on both.

Re: Why is my alert triggering false positives?

nmayafit — Thu, 20 Apr 2017 15:24:54 GMT

Update: I found that the problem is that I used a real time schedule. When I changed it to cron schedule (every min) everything started working fine.

Re: Why is my alert triggering false positives?

aaraneta_splunk — Thu, 20 Apr 2017 18:49:53 GMT

@nmayafit - Glad to hear you found the solution to your question. Please don't forget to click "Accept" to resolve your post so that others can easily find it. Thanks!

Re: Why is my alert triggering false positives?

bishtk — Wed, 31 Oct 2018 09:17:26 GMT

Hi Techies,

I am facing same issue and with many alerts. But I don't have any of them set to Real Time. All are set to cron only.