Splunk Search

delta then sum then graph from multiple hosts

charleswheelus
Path Finder

I have log entries from multiple hosts which contain cumulative counters. One log entry per host about every 5 minutes. Each counter on each host has a value unique to that host. Something like this:

2012-05-02T04:28:54+00:00 cp2.prod.b.example.com XACT=31568

2012-05-02T04:28:32+00:00 cp1.prod.a.example.com XACT=22622

2012-05-02T04:28:25+00:00 cp2.prod.a.example.com XACT=24623

2012-05-02T04:28:21+00:00 cp1.prod.b.example.com XACT=31140

2012-05-02T04:23:54+00:00 cp2.prod.b.example.com XACT=31500

2012-05-02T04:23:32+00:00 cp1.prod.a.example.com XACT=22600

2012-05-02T04:23:25+00:00 cp2.prod.a.example.com XACT=24600

2012-05-02T04:23:21+00:00 cp1.prod.b.example.com XACT=31100



I would like to be able to:

1) get the deltas between XACT in log entries on a per host basis

2) sum those deltas

3) graph the sum of the deltas over time (timechart)

gkanapathy
Splunk Employee
Splunk Employee

Well, first, I'm not totally certain that you really need to compute and sum every delta -- if they're simply cumulative counters, couldn't you just subtract the first value from the last (within each time interval and for each host)? That would be more efficient.

... | bucket _time span=10min 
    | stats earliest(XACT) as begin
            latest(XACT) as end
      by _time, host
    | eval delta=end-begin
    | xyseries _time,host,delta

or (less preferred):

... | bucket _time span=10min 
    | stats earliest(XACT) as begin
            latest(XACT) as end
      by _time, host
    | eval delta=end-begin
    | timechart span=10m sum(delta) as delta by host

But for the curious, if you really wanted to do it the other way (again, the above will be faster, especially so if you have multiple indexers), you can use streamstats:

... | streamstats current=t global=f window=2
            earliest(XACT) as curr
            latest(XACT) as next
      by host
    | eval delta=next-curr
    | timechart span=10m sum(delta) as delta by host

woodcock
Esteemed Legend

Actually this would be a bit more RAM efficient:

 ... | streamstats current=t global=f window=2 range(XACT) AS delta BY host
 | timechart span=10m sum(delta) AS delta BY host

charleswheelus
Path Finder

The last option provided the results I needed.

Thanks for the awesome answer!

0 Karma

fabiocaldas
Contributor

Those streamstats tips is awesome

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...