Splunk Search

How to correctly use mstats rate() without listing all dimensions in BY clause?

y0ft
Explorer

I have a monotonic counter metric named http_req and my metric data points are tagged with path and host dimensions, eg. path=/accounts or path=/transactions, host=server-1 or host=server-2.

Application running on each host starts with metric value 0 and increases the value after every request processed. If host is restarted, counting restarts from 0. Every application creates it's own metric data points.

I can create a visualization showing number of calls processed in time while grouping per path and discarding the host aspect using:

| mstats rate(http_req) as http_req_rate WHERE index="metrices" by path,host span=1m
| stats sum(http_req_rate) by _time,path
| xyseries _time path sum(http_req_rate)

I noticed that I need to specify all dimensions in BY clause of mstats for rate() to work correctly. Having known set of dimensions I can create proper query, but in the future more dimensions will be added.

Can this query be written in such way that listing all the dimensions is not necessary? eg. is there something like presented below possible?

| mstats ... by all-dimensions
0 Karma
1 Solution

y0ft
Explorer

I found a solution:

| mstats rate(http_req) as http_req_rate WHERE index="metrices" by _timeseries,path span=1m
| stats sum(http_req_rate) by _time,path
| xyseries _time path sum(http_req_rate)

Documentation around it could be improved. It is only mentioned in https://docs.splunk.com/Documentation/Splunk/8.0.1/Metrics/Histogramdatatype

View solution in original post

0 Karma

y0ft
Explorer

I found a solution:

| mstats rate(http_req) as http_req_rate WHERE index="metrices" by _timeseries,path span=1m
| stats sum(http_req_rate) by _time,path
| xyseries _time path sum(http_req_rate)

Documentation around it could be improved. It is only mentioned in https://docs.splunk.com/Documentation/Splunk/8.0.1/Metrics/Histogramdatatype

0 Karma

y0ft
Explorer

I found a workaround: add another dimension - tsid (time series identifier) - with random constant value for every counter.

| mstats rate(http_req) as http_req_rate WHERE index="metrices" by tsid,path span=1m
| stats sum(http_req_rate) by _time,path
| xyseries _time path sum(http_req_rate)
0 Karma
Get Updates on the Splunk Community!

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...

Updated Team Landing Page in Splunk Observability

We’re making some changes to the team landing page in Splunk Observability, based on your feedback. The ...