Splunk Search

How to suppress first DELTA value?

anmcgill
Loves-to-Learn Lots

I am trying to alert on any processes where their CPU time is gaining 60 sec for every elapsed minute.  I am using the following search to calculate the delta in CPU time per process:

... | stats max(cpu_time_sec) as maxTime by PID _time
| delta maxTime as deltaTime 
| table _time PID deltaTime

I get the following output as an example (the above was filtered on a specific PID to get the below output):

_timePIDmaxTimedeltaTime
2021-03-29 13:28:44PID: 2691628572856
2021-03-29 13:29:45PID: 2691628570
2021-03-29 13:30:44PID: 2691628570
2021-03-29 13:31:45PID: 2691628570

 

The first value is always higher than it "should" be as the CPU time at that point is compared to a non-existent previous interval value unless I select a large enough time range that it predates when the process started.  I do not want to do this.

I am only strictly interested in getting the most recent deltas for the monitored processes and flagging those that have 60 sec of accumulated CPU time.  If I put a  where deltaTime>60 on my query I erroneously capture the first entry.

Any insights on how to accomplish this?

Labels (1)
0 Karma

bowesmana
SplunkTrust
SplunkTrust

Very simply you could add this after your table command

| streamstats c by PID
| where c>1

 which will exclude the first event for all PID.

 

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Observability Simplified: Combining User Experience, Application Performance & ...

Tech Talk Observability Simplified: Combining User Experience, Application Performance & Network ...

Event Series May & June: From Network Visibility to Service Intelligence

Unifying the Network: Moving from Alert Noise to Service Intelligence with Splunk ITSI In today’s hybrid ...

Global Splunk User Group Events: May + June 2026

Your Splunk Community Awaits: Discover Upcoming User Group Events Worldwide    Staying ahead in the fast-paced ...