topic How to correctly use mstats rate() without listing all dimensions in BY clause? in Splunk Search

How to correctly use mstats rate() without listing all dimensions in BY clause?

y0ft — Tue, 21 Jan 2020 12:38:24 GMT

I have a monotonic counter metric named http_req and my metric data points are tagged with path and host dimensions, eg. path=/accounts or path=/transactions, host=server-1 or host=server-2.

Application running on each host starts with metric value 0 and increases the value after every request processed. If host is restarted, counting restarts from 0. Every application creates it's own metric data points.

I can create a visualization showing number of calls processed in time while grouping per path and discarding the host aspect using:

| mstats rate(http_req) as http_req_rate WHERE index="metrices" by path,host span=1m
| stats sum(http_req_rate) by _time,path
| xyseries _time path sum(http_req_rate)

I noticed that I need to specify all dimensions in BY clause of mstats for rate() to work correctly. Having known set of dimensions I can create proper query, but in the future more dimensions will be added.

Can this query be written in such way that listing all the dimensions is not necessary? eg. is there something like presented below possible?

| mstats ... by all-dimensions

Re: How to correctly use mstats rate() without listing all dimensions in BY clause?

y0ft — Mon, 27 Jan 2020 09:12:34 GMT

I found a workaround: add another dimension - tsid (time series identifier) - with random constant value for every counter.

| mstats rate(http_req) as http_req_rate WHERE index="metrices" by tsid,path span=1m
| stats sum(http_req_rate) by _time,path
| xyseries _time path sum(http_req_rate)

Re: How to correctly use mstats rate() without listing all dimensions in BY clause?

y0ft — Mon, 27 Jan 2020 13:55:44 GMT

I found a solution:

| mstats rate(http_req) as http_req_rate WHERE index="metrices" by _timeseries,path span=1m
| stats sum(http_req_rate) by _time,path
| xyseries _time path sum(http_req_rate)

Documentation around it could be improved. It is only mentioned in https://docs.splunk.com/Documentation/Splunk/8.0.1/Metrics/Histogramdatatype