I am trying to alert on any processes where their CPU time is gaining 60 sec for every elapsed minute. I am using the following search to calculate the delta in CPU time per process:
... | stats max(cpu_time_sec) as maxTime by PID _time
| delta maxTime as deltaTime
| table _time PID deltaTime
I get the following output as an example (the above was filtered on a specific PID to get the below output):
| _time | PID | maxTime | deltaTime |
| 2021-03-29 13:28:44 | PID: 26916 | 2857 | 2856 |
| 2021-03-29 13:29:45 | PID: 26916 | 2857 | 0 |
| 2021-03-29 13:30:44 | PID: 26916 | 2857 | 0 |
| 2021-03-29 13:31:45 | PID: 26916 | 2857 | 0 |
The first value is always higher than it "should" be as the CPU time at that point is compared to a non-existent previous interval value unless I select a large enough time range that it predates when the process started. I do not want to do this.
I am only strictly interested in getting the most recent deltas for the monitored processes and flagging those that have 60 sec of accumulated CPU time. If I put a where deltaTime>60 on my query I erroneously capture the first entry.
Any insights on how to accomplish this?
