Splunk Observability Cloud

APM -http.server.request.duration metrics coming as grouped instead of Histogram type

Maheswari1812
Explorer
http.server.request.durationhistogramDuration of HTTP server requests.

metrics coming as grouped like below

  • http.server.request.duration_sum
    http.server.request.duration_count
    http.server.request.duration_max
    http.server.request.duration_bucket
    http.server.request.duration_min
  • http.client.request.duration_count similarly...

http.route as well coming as Gain/Vl/* instead of full end point. Any solution for this. 

 

0 Karma

Maheswari1812
Explorer

Splunk Observability endpoint grouping settings adjustment
http.route coming as A/* - grouped endpoint
but url.path coming as A/B/C/D - full endpoint

how to fix this issue ? Anyone can help?

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

What is your search/SPL for this?

0 Karma

Maheswari1812
Explorer

A = data('http.server.request.duration_min', filter=filter('http.route', '/Gain/vl/*')).publish(label='A') -signalflow builder query.

 

I need histogram function metrics instead of data. Is there any configuration changes required as part of otel instrumentation.

A = histogram('http.server.request.duration', filter=filter('http.route', '/batch-process/iomatch')).min().publish(label='A')

 

 

0 Karma

premadhas
Engager

Hi, @Maheswari1812 You was able to figureout the solution for this? If so can you please share?

I am also experiencing the same issue.
In my case, I want to set up a monitor to track latency for individual endpoints. However, when I search for the metric http.server.request.duration in the Observability Metric Finder, I see it listed (as shown in the screenshot), but when I click on it, it does not show the direct metric data. Instead, it only displays the aggregated metrics such as sum, count, min, max, and bucket, which are derived from http.server.request.duration.

My requirement is to access http.server.request.duration itself so that I can configure a monitor based directly on the latency duration. Could you please guide me on how to access this metric or suggest the correct metric to use for endpoint-level latency monitoring?

Screenshot 2025-11-27 at 02.31.13.pngScreenshot 2025-11-27 at 03.29.12.png

0 Karma

CesarF
Splunk Employee
Splunk Employee

I appreciate this is a (very) late reply! The signalfx exporter breaks down the histogram into their aggregates metrics by default (backward compability).

Use the parameter send_otlp_histograms on the signalfx exporter config and you will get the proper histogram.

https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/signalfxexporte...

Tags (1)
0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

[Puzzles] Solve, Learn, Repeat: Character substitutions with Regular Expressions

This challenge was first posted on Slack #puzzles channelFor BORE at .conf23, we had a puzzle question which ...

Splunk Community Badges!

  Hey everyone! Ready to earn some serious bragging rights in the community? Along with our existing badges ...

[Puzzles] Solve, Learn, Repeat: Matching cron expressions

This puzzle (first published here) is based on matching timestamps to cron expressions.All the timestamps ...