I'm trying to write a search which can detect the occurrence of an event AFTER a previous event containing the same field value has occurred. The use case is that when the first event occurs, we trigger an alert (which has a subsequent course of action internally) and when the subsequent log event occurs matching the same ID, we know that the issue has been fixed.
Ideally, we'd like for Splunk to search for the occurrence of the second event from the time the first event occurred. So for every device ID in event 1, look for a subsequent event 2 since the first event occurred, and trigger an alert saying "for this device ID, the second event has happened".
Couple of notes:
- The device ID field name is different in the two events, I have previously corrected it using an eval, for example:
search < event1> | eval deviceID = devID
The second event always occurs AFTER the first event. And the second event is basically indicating that a device has been registered, we have a thousand registrations a day, and it's hard to specify a time modifier. So either I could go with something generic and say "search in the last 2 weeks if you found a registration for devID" or I could say "search from earliest="when the first event happened" to now for the second event"
Things I've tried so far:
event 1 OR event 2 | eval deviceID = devID | transaction deviceID | event 1 AND event 2
I tried to put both events in a transaction and say only show me results where BOTH events are available so I can alert on that. However, this doesn't seem to work as expected perhaps because of the eval to change the field name?
Also, to specify time range. I tried putting the timestamp of event 1 into a field (say "first_event_start") using eval and then used a join to look for the second event starting earliest=first_event_start, but I got an error (looks like it only takes numeric values or time modifiers).