Splunk Search

Summary stats from distinct (overflowing) counters.

PickleRick
SplunkTrust
SplunkTrust

Hello.

I have a set of hosts which send some stats. In my case these are rsyslog impstats statistics but it can be anything - for example SNMP interface counters.

The point is that I have a counter which increases with time and I want to compute incremental statistics.

Yes, I know you'll point me towards delta command but it can only count difference from one even to another and I have several different sources for which I need separate stat. (let's say something like | delta <parameter> by host - unfortunately there's no such command ;-)).

After some poking around it seems that range() statistical function seems to fit nicely - it calculates - as the name implies - a range  between lowest and highest  value of the given field so if I pair it with timechart it works beautifuly. Almost.

The problem is that the counters have finite length and after some time overflow back to 0.  And if this happens... of course the range() command returns some ridiculous values.

If it was a simple delta calculation, I'd probably just do some modulo operation or some other conditional eval to account for it but I don't see a reasonable way to do it with already summed up  values since even the field names of the summary table are variable and depend on host names and I can't know the list of hosts beforehand.

Is there any reasonable way to filter out the "overflowed" values? Just using "outliers" removes also "bottom" values which is not what I need.

Labels (1)
Tags (3)
0 Karma
Get Updates on the Splunk Community!

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...

Adoption of Infrastructure Monitoring at Splunk

  Splunk's Growth Engineering team showcases one of their first Splunk product adoption-Splunk Infrastructure ...