Generate alert when 2 consecutive events occurred for each server

[acirulli]

I have several servers sending me log.

For each log I have a field called X if X=100 for two consecutive times I have to generate an alert.

I would like to produce 1 query to monitor every server and I want it to generate an alert if one of the servers has 2 consecutive events (described above) occurring.

Any idea?

[Richfez]

Try using streamstats with a two-event window, then search where the total is equal to 200.

... | streamstats window=2 sum(X) as Totals by host | search Totals=200

[acirulli]

Seems working.
But I have 142 consecutive events with x=100 if I ran your query I got 9 events, I would expect 71 (a half), am I wrong?

If i ran the same query with window 1 and filter for 100 I got 142 events and this is what I expect.

[Richfez]

Do you have a few sample events? That might help.

A couple of notes, though - make sure splitting by "host" is correct, I just threw that in there as "host" but you asked for "server" (I assumed they are the same, but I could be wrong). If your 142 are from different servers, it wouldn't trigger this each time. Also make sure you aren't trimming out the X!=100 events ahead of time, this requires them to be there in order to make it not spam you (i.e. your base search should be fairly wide open and not specify X=anything.) As always, the devil is in the details!

BTW, with your example IF those all came from the same server, you'd actually have 141 triggered alerts. When event 1 comes in it won't trigger because there's only one event. When event two comes in with X=100, it'll trigger on those two. When event three comes in it'll trigger on event 2 and 3. When event four comes in it'll trigger on events 3 and 4.

If you'd like different behavior, we can do other things too. Keep in mind for an ALERT you can throttle it, too.

[acirulli]

Host is right and I'm not trimming any value:

Here is my query:
index=xxx sourcetype=xxxx | streamstats window=2 sum(PLOSS_PING) as total by host | search total=200

Everything seems fine to me but the output is not the one expected....

In this image I provide you an output example of the query:
https://unsee.cc/togedupi/

In this other one an example of series of consecutive events that did not show up in the query

https://unsee.cc/gubamote/

What do you think?

To me something is wrong...

[Richfez]

Ah,

Add global=f to the streamstats.

index=xxx sourcetype=xxxx | streamstats window=2 sum(PLOSS_PING) as total by host global=f| search total=200 

(Did I mention I apparently don't use the by clause much?)

[acirulli]

Version is 6.3

[acirulli]

The same 9 results instead of 141 😞

[acirulli]

Is it considering two consecutive in general or two consecutive by the same host?