How to create an alert to trigger after multiple events with similar data are detected over a period of time?

[horsefez]

Hi,

I'm trying to design real time alerts who trigger if "one" or more events with similar event properties are detected over a period of time.

Imagine data like this:

TIME                  Action    IP
08-10-2015 02:10:00   ABCDE     192.168.1.1
08-10-2015 02:15:00   ABCDE     192.168.1.1
08-10-2015 02:25:00   ABCDE     192.168.1.1
08-10-2015 02:30:00   ABCDE     192.168.1.2
08-10-2015 02:35:00   ABCDE     192.168.1.1

This search string:

sourcetype=imaginarydatasource | transaction maxspan=20min IP

would give back these results:

Event 1
    08-10-2015 02:10:00   ABCDE  192.168.1.1
    08-10-2015 02:15:00   ABCDE  192.168.1.1
    08-10-2015 02:25:00   ABCDE  192.168.1.1
Event 2
    08-10-2015 02:30:00   ABCDE  192.168.1.2
Event 3
    08-10-2015 02:35:00   ABCDE  192.168.1.1

Is it possible to create a working alert, that searches data in real-time with this search string and then sends me grouped up event data like in this example via email?
(3 emails in this example)
Or should I schedule a hourly alert?

Thanks in advance!

[amnonh]

I honestly don't get why this is marked as solved. I see a very clear question and the marked "solution" only brings up more questions than it answers.

Nothing in this post helps me in achieving a working end result as and answer to the clear question. 😕

[Richfez · Accepted solution]

Real-time is a big gotcha here. Given your example, how would the transaction command know when a transaction is closed? If you were watching those events in real time, what would have prevented the alert from triggering at the first item at 2:10? At 2:10:01 that event is in Splunk and a RT search would have seen it and it would have triggered your alert.

Because of that, a scheduled report may be better.

Of course there's a but. If you had a good closing event for those transactions, you could add an endswith=X clause to the transaction to prevent closing transactions until there's an actual closing event. This would work better for schedules searches even on a 1, 5 or 15 minute interval, and might let it work fairly well as a real time alert*.

And another but: Think hard before doing this as real time. a) It's expensive. b) It will likely become annoying the third time it happens and c) what would you actually do in real time that having it tell you once per minute or every 5 minutes wouldn't do? IMO, given that I know nothing about any use case behind this question, I *might schedule at most a once-per-hour report that summarizes this information (number of transactions that happened, etc...) on a custom schedule during normal work hours. I would probably send an email once per day to a few interested parties and make sure a link to the report or dashboard is enclosed. Use the dashboard for your look at "current" information.

[MuS]

+1 for scheduled alert over the last 2 or 5 minutes, so you will get those late arriving events as well.

[horsefez]

Thank you rich7177!
I thought of it in a similar way shortly after posting the question. It indeed isn't practicable to do this in real time.
Thanks to you I now have a better understanding of it. 🙂

The endswith parameter is a good hint, I'll see what I can achieve with it. Thanks for that aswell!