I'm trying to write a search which can detect the occurrence of an event AFTER a previous event containing the same field value has occurred. The use case is that when the first event occurs, we trigger an alert (which has a subsequent course of action internally) and when the subsequent log event occurs matching the same ID, we know that the issue has been fixed.
Event 1: <time=11/2/2017 11:00:00> sourcetype=firstevent "Event 1 happened" devID=ABCD | ... Event 2: <time=11/3/2017 02:00:00> sourcetype=secondevent "Event 2 happened" deviceID=ABCD | ...
Ideally, we'd like for Splunk to search for the occurrence of the second event from the time the first event occurred. So for every device ID in event 1, look for a subsequent event 2 since the first event occurred, and trigger an alert saying "for this device ID, the second event has happened".
Couple of notes:
- The device ID field name is different in the two events, I have previously corrected it using an eval, for example:
search < event1> | eval deviceID = devID
Things I've tried so far:
event 1 OR event 2 | eval deviceID = devID | transaction deviceID | event 1 AND event 2
I tried to put both events in a transaction and say only show me results where BOTH events are available so I can alert on that. However, this doesn't seem to work as expected perhaps because of the eval to change the field name?
Also, to specify time range. I tried putting the timestamp of event 1 into a field (say "firsteventstart") using eval and then used a join to look for the second event starting earliest=firsteventstart, but I got an error (looks like it only takes numeric values or time modifiers).
Any help would be great! Thanks!
have you tried using transaction with the
endswith parameters yet?
Try something like this:
<yoursearch> | transaction deviceID startswith="<indicator that event 1 happened>" endswith="<indicator that event 2 happened>"
The indicator I'm talking about is some sort of raw string that marks the event