delta then sum then graph from multiple hosts

[charleswheelus]

I have log entries from multiple hosts which contain cumulative counters. One log entry per host about every 5 minutes. Each counter on each host has a value unique to that host. Something like this:

2012-05-02T04:28:54+00:00 cp2.prod.b.example.com XACT=31568

2012-05-02T04:28:32+00:00 cp1.prod.a.example.com XACT=22622

2012-05-02T04:28:25+00:00 cp2.prod.a.example.com XACT=24623

2012-05-02T04:28:21+00:00 cp1.prod.b.example.com XACT=31140

2012-05-02T04:23:54+00:00 cp2.prod.b.example.com XACT=31500

2012-05-02T04:23:32+00:00 cp1.prod.a.example.com XACT=22600

2012-05-02T04:23:25+00:00 cp2.prod.a.example.com XACT=24600

2012-05-02T04:23:21+00:00 cp1.prod.b.example.com XACT=31100



I would like to be able to:

1) get the deltas between XACT in log entries on a per host basis

2) sum those deltas

3) graph the sum of the deltas over time (timechart)

[gkanapathy]

Well, first, I'm not totally certain that you really need to compute and sum every delta -- if they're simply cumulative counters, couldn't you just subtract the first value from the last (within each time interval and for each host)? That would be more efficient.

... | bucket _time span=10min 
    | stats earliest(XACT) as begin
            latest(XACT) as end
      by _time, host
    | eval delta=end-begin
    | xyseries _time,host,delta

or (less preferred):

... | bucket _time span=10min 
    | stats earliest(XACT) as begin
            latest(XACT) as end
      by _time, host
    | eval delta=end-begin
    | timechart span=10m sum(delta) as delta by host

But for the curious, if you really wanted to do it the other way (again, the above will be faster, especially so if you have multiple indexers), you can use streamstats:

... | streamstats current=t global=f window=2
            earliest(XACT) as curr
            latest(XACT) as next
      by host
    | eval delta=next-curr
    | timechart span=10m sum(delta) as delta by host

[woodcock]

Actually this would be a bit more RAM efficient:

 ... | streamstats current=t global=f window=2 range(XACT) AS delta BY host
 | timechart span=10m sum(delta) AS delta BY host

[charleswheelus]

The last option provided the results I needed.

Thanks for the awesome answer!

[fabiocaldas]

Those streamstats tips is awesome