Monitoring Splunk

How to find delta of cumulative values by day?

Engager

I am storing some performance counters in splunk. The data gets written one or more times per day (though not necessarily at the exactly the same time each day) and the totals are cumulative for multiple days.

For example, the data may include "total_worker_time" for various stored procedures which we can assume will be an ever-increasing value. I would like to chart the change in this value over time per stored procedure to show how much work time was spent on a particular day for each stored procedure. For each day, I want to find the maximum value by subtracting the maximum previous day's value to find the delta.

I know I can do:

`````` <...mysearch...>| timechart max(total_worker_time)  span=1d  by procname
``````

to return the cumulative total for each procname like this:

``````_time   Proc1   Proc2
2018-01-16  29710092875 4354587351
2018-01-17  54315798685 5977664529
2018-01-18  78055137053 7739773570
``````

but I'd rather return the difference like this:

``````DATE    Proc1Delta  Proc2Delta
1/16/2018   no prev value   no prev value
1/17/2018   24605705810 1623077178
1/18/2018   23739338368 1762109041
``````

Is this possible?

Tags (5)
1 Solution
Champion

Well, drat. If you only have two `procname` values, then it's not too bad to do it like this, I guess:

``````<...mysearch...>
| timechart max(total_worker_time) span=1d procname
| delta Proc1 AS Proc1Delta p=1
| delta Proc2 AS Proc2Delta p=1
``````

Does that do it?

Champion

Well, drat. If you only have two `procname` values, then it's not too bad to do it like this, I guess:

``````<...mysearch...>
| timechart max(total_worker_time) span=1d procname
| delta Proc1 AS Proc1Delta p=1
| delta Proc2 AS Proc2Delta p=1
``````

Does that do it?

Champion

Looks like something the Splunk command `delta` would be good for doing. How about this:

``````<...mysearch...>
| timechart max(total_worker_time)  span=1d  by procname
| foreach Proc*
[ |delta <<FIELD>> AS <<FIELD>>Delta p=1 ]
``````

I don't have access to my system at the moment to test this, so I'm air-coding. Let me know if this works; if not, I'll try to help tweak it.

Engager

Thanks for your help. When I run it exactly as you have above I get an error:

Error in 'foreach' command: Search pipeline may not contain non-streaming commands

Engager

That will do it! Would be nice to be able to pivot over the names, but I can work with this. Thanks again!

Champion

Yeah, I didn't think about the fact that `delta` isn't a steaming command. It looks so much nicer all bundled up inside a foreach loop! 🙂

Get Updates on the Splunk Community!

Splunk Security Content for Threat Detection & Response, Q1 Roundup

Join Principal Threat Researcher, Michael Haag, as he walks through:An introduction to the Splunk Threat ...

Splunk Life | Happy Pride Month!

Happy Pride Month, Splunk Community! &#x1f308; In the United States, as well as many countries around the ...

SplunkTrust | Where Are They Now - Michael Uschmann

The Background Five years ago, Splunk published several videos showcasing members of the SplunkTrust to share ...