How do you search metrics? (contradictory documentation)

[sillingworth]

The documentation appears to contradict itself on this. The mstats documentation tends to perform its functions on the metric name (eg "avg(this_is_my_metric_name)") and states:

The WHERE clause cannot filter by metric_name. metric_name filtering is performed based on the metric_name fields specified by the <stats-metric-term> argument.

but the Metrics documentation does the opposite in its examples - its stats-metric-term is "_value" rather than a metric name, and its where clause filters on the metric name:

| mstats avg(_value) WHERE index=mymetricdata AND metric_name=aws.ec2.CPUUtilization span=30s

So which is the truth, and why do the documents contradict each other, or am I misunderstanding something?

[thaggie_splunk · Accepted solution]

mstats can be used in two ways, where the metric name is used in the aggregation and where it is used in the filter.
To group by time, you add a span.

These are equivalent:

| mstats avg(_value) WHERE metric_name="my.metric.name" AND index="my-metrics-index" span=10s
| mstats avg(my.metric.name) WHERE index="my-metrics-index" span=10s

[sillingworth]

So I should never put "by _time" - simply including a span does that automatically?

[thaggie_splunk]

That is correct

[thaggie_splunk]

A good way to learn the mstats command is to install the "Metrics Workspace" app https://splunkbase.splunk.com/app/4192/ and see what it does.

[inventsekar]

Good learning about "metrics" feature. thanks for sharing.

[sillingworth]

Great - I'll do that, as I'm still struggling to do anything more than a simple search (eg a time chart by host). Thanks.

[sillingworth]

mstats documentation: http://docs.splunk.com/Documentation/Splunk/7.2.0/SearchReference/Mstats
Metrics documentation: https://docs.splunk.com/Documentation/Splunk/7.2.0/Metrics/Search

[sillingworth]

Also, does anybody have an example of grouping by _time that actually works? I always get "Error in 'mstats' command: Metrics Processor: Specify the span argument to group events by time." even after including a span.