Splunk Search

Summary stats from distinct (overflowing) counters.

PickleRick
SplunkTrust
SplunkTrust

Hello.

I have a set of hosts which send some stats. In my case these are rsyslog impstats statistics but it can be anything - for example SNMP interface counters.

The point is that I have a counter which increases with time and I want to compute incremental statistics.

Yes, I know you'll point me towards delta command but it can only count difference from one even to another and I have several different sources for which I need separate stat. (let's say something like | delta <parameter> by host - unfortunately there's no such command ;-)).

After some poking around it seems that range() statistical function seems to fit nicely - it calculates - as the name implies - a range  between lowest and highest  value of the given field so if I pair it with timechart it works beautifuly. Almost.

The problem is that the counters have finite length and after some time overflow back to 0.  And if this happens... of course the range() command returns some ridiculous values.

If it was a simple delta calculation, I'd probably just do some modulo operation or some other conditional eval to account for it but I don't see a reasonable way to do it with already summed up  values since even the field names of the summary table are variable and depend on host names and I can't know the list of hosts beforehand.

Is there any reasonable way to filter out the "overflowed" values? Just using "outliers" removes also "bottom" values which is not what I need.

Labels (1)
Tags (3)
0 Karma
Get Updates on the Splunk Community!

Introducing Splunk Enterprise 9.2

WATCH HERE! Watch this Tech Talk to learn about the latest features and enhancements shipped in the new Splunk ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...