Splunk Search

Working with OpenTelemetry Cumulative Histogram Bucket with Calculations

cchan
Loves-to-Learn

Hi, I am new to Splunk and couldn't figure out how to work with OpenTelemetry's histogram bucket in Splunk. 

I have a basic set up of 3 buckets from OTel, with le=2000, 8000, +Inf and the bucket name is "http.server.duration_bucket".

My goal is to display the number count inside the 3 buckets for a 15min period, perform a calculations using those values, and add the calculated value as a 4th column.

I came up with this so far:

 

 

 

| mstats max("http.server.duration_bucket") chart=true WHERE "index"="metrics" span=15m BY le
| fields - _span*
| rename * AS "* /s"
| rename "_time /s" AS _time

 

 

 

But immediately I see 2 issues:

a) the 8000 bucket results are added with 2000 bucket results as well because they are recorded as cumulative histograms.

b) the values inside the bucket is always increasing, so I cannot isolate how many counts belong to 2000 bucket now vs the same bucket 15mins ago.

And I realized that I don't know how to get the right calculation and separate the buckets without using "BY le", so I cannot perform calculations from there.

So my question is:

1) Is there an example of function for displaying the real non-cumulative values in the histogram for a given period?

2) If my calculation is max(le=2000)*0.6 + max(le=8000)*0.4, how would I add that as a column to the search?

Thanks in advance!

 

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Take Action Automatically on Splunk Alerts with Red Hat Ansible Automation Platform

 Are you ready to revolutionize your IT operations? As digital transformation accelerates, the demand for ...

Calling All Security Pros: Ready to Race Through Boston?

Hey Splunkers, .conf25 is heading to Boston and we’re kicking things off with something bold, competitive, and ...

Beyond Detection: How Splunk and Cisco Integrated Security Platforms Transform ...

Financial services organizations face an impossible equation: maintain 99.9% uptime for mission-critical ...