Splunk Search

Working with OpenTelemetry Cumulative Histogram Bucket with Calculations

cchan
Loves-to-Learn

Hi, I am new to Splunk and couldn't figure out how to work with OpenTelemetry's histogram bucket in Splunk. 

I have a basic set up of 3 buckets from OTel, with le=2000, 8000, +Inf and the bucket name is "http.server.duration_bucket".

My goal is to display the number count inside the 3 buckets for a 15min period, perform a calculations using those values, and add the calculated value as a 4th column.

I came up with this so far:

 

 

 

| mstats max("http.server.duration_bucket") chart=true WHERE "index"="metrics" span=15m BY le
| fields - _span*
| rename * AS "* /s"
| rename "_time /s" AS _time

 

 

 

But immediately I see 2 issues:

a) the 8000 bucket results are added with 2000 bucket results as well because they are recorded as cumulative histograms.

b) the values inside the bucket is always increasing, so I cannot isolate how many counts belong to 2000 bucket now vs the same bucket 15mins ago.

And I realized that I don't know how to get the right calculation and separate the buckets without using "BY le", so I cannot perform calculations from there.

So my question is:

1) Is there an example of function for displaying the real non-cumulative values in the histogram for a given period?

2) If my calculation is max(le=2000)*0.6 + max(le=8000)*0.4, how would I add that as a column to the search?

Thanks in advance!

 

Labels (1)
0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...