Situation: I have jobs that start running at different times because they are dependent on previous jobs to run successfully. There are two events I am concerned with. One event for "job started" and another event for "job finished".
Goal: Alert when the job has been running for 5 hours or more, regardless of when it started, and has not finished
My approach was to calculate the duration based on the _time
from "job started" but I can't determine the syntax on how to prevent an alert if the job finished on time since the "job finished" _time
is in a separate event.
I am able to determine the duration using | eval secondsAgoStr=tostring(now() - _time, "duration"
I am also able to calculate the delta of the time using | delta _time AS timeDeltaSeconds p=1
I can't use either of these when the job is currently running because the "job finished" event has not occurred.
Assuming your events have a field to uniquely identify a job, by either a jobID or jobName field, you can do like this.
your current search to fetch both "job started" and "job finished" events
| eval eventType=if(searchmatch("job started"),"Start","End")
| chart values(_time) over jobID by eventType
| eval End=coalesce(End,now())
| eval jobduration=End-Start | where jobduration>=18000
This helped me come up with the alert. Thanks
Assuming your events have a field to uniquely identify a job, by either a jobID or jobName field, you can do like this.
your current search to fetch both "job started" and "job finished" events
| eval eventType=if(searchmatch("job started"),"Start","End")
| chart values(_time) over jobID by eventType
| eval End=coalesce(End,now())
| eval jobduration=End-Start | where jobduration>=18000