All Apps and Add-ons

Lower resolution chart when reproducing grafana/prometheus in Splunk with same data

sadohert
New Member

I'm trying to reproduce a collection of Grafana dashboards we have using Splunk. Generally its working okay, but there's this one area I can't get my head around. I seem to get better resolution on the charts with prometheus/Grafana than I see with Splunk.

In Grafana I have a chart that looks like this:
alt text

Where the prometheus query is:
irate(mattermost_http_errors_total{instance=~"$server"}[1m])

In Splunk my chart looks like this:
alt text

And the associated Splunk search is:

| mstats rate(_value) as count prestats=true WHERE metric_name="mattermost_http_errors_total" AND `mattermost_metrics` sourcetype=prometheus:metric AND (host ="617d67c7fe56") span=60s BY host
| timechart rate(_value) as count span=60s BY host
| addtotals

The general structure/trends seem reasonably close, but Grafana seems to have much finer resolution on the data.

  • I'm scraping the system-under-test in 15s intervals in both cases.
  • Hovering over the datapoints shows me that Grafana is generating a value on the same period as the scraping interval, but Splunk is only generating data on the span value.

What I suspect: Its as if Grafana is doing a sliding 1 minute window, evaluated at each point in time that it has data. Where Splunk is taking absolute windows/span and generating a single result.

Is there something I'm missing with how to author the mstats searches? Is there some way to achieve a similar result? Am I correct in what I suspect is happening?

Note: My Grafana dashboard also has a series for the "Total" in the same way my Splunk query suggests with addtotals... I just didn't copy that query here

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...