I am storing some performance counters in splunk. The data gets written one or more times per day (though not necessarily at the exactly the same time each day) and the totals are cumulative for multiple days.
For example, the data may include "total_worker_time" for various stored procedures which we can assume will be an ever-increasing value. I would like to chart the change in this value over time per stored procedure to show how much work time was spent on a particular day for each stored procedure. For each day, I want to find the maximum value by subtracting the maximum previous day's value to find the delta.
I know I can do:
<...mysearch...>| timechart max(total_worker_time) span=1d by procname
to return the cumulative total for each procname like this:
_time Proc1 Proc2
2018-01-16 29710092875 4354587351
2018-01-17 54315798685 5977664529
2018-01-18 78055137053 7739773570
but I'd rather return the difference like this:
DATE Proc1Delta Proc2Delta
1/16/2018 no prev value no prev value
1/17/2018 24605705810 1623077178
1/18/2018 23739338368 1762109041
Is this possible?
Well, drat. If you only have two procname
values, then it's not too bad to do it like this, I guess:
<...mysearch...>
| timechart max(total_worker_time) span=1d procname
| delta Proc1 AS Proc1Delta p=1
| delta Proc2 AS Proc2Delta p=1
Does that do it?
Well, drat. If you only have two procname
values, then it's not too bad to do it like this, I guess:
<...mysearch...>
| timechart max(total_worker_time) span=1d procname
| delta Proc1 AS Proc1Delta p=1
| delta Proc2 AS Proc2Delta p=1
Does that do it?
Looks like something the Splunk command delta
would be good for doing. How about this:
<...mysearch...>
| timechart max(total_worker_time) span=1d by procname
| foreach Proc*
[ |delta <<FIELD>> AS <<FIELD>>Delta p=1 ]
I don't have access to my system at the moment to test this, so I'm air-coding. Let me know if this works; if not, I'll try to help tweak it.
Thanks for your help. When I run it exactly as you have above I get an error:
Error in 'foreach' command: Search pipeline may not contain non-streaming commands
That will do it! Would be nice to be able to pivot over the names, but I can work with this. Thanks again!
Yeah, I didn't think about the fact that delta
isn't a steaming command. It looks so much nicer all bundled up inside a foreach loop! 🙂