topic Re: Using mstats with eval grouped by field in Splunk Search

Using mstats with eval grouped by field

prandelicious — Thu, 23 Jul 2020 10:56:02 GMT

I wanted to graph the computed value of two fields and group the result by another field:

| mstats avg(kube.pod.cpu.limit) AS cpu_limit avg(kube.pod.cpu.usage_rate) AS cpu_usage WHERE index="metrics" span=auto BY "pod-name" | eval utilization=((cpu_usage/cpu_limit) * 100) | timechart values(utilization) agg=max limit=5 useother=false BY "pod-name" | fields - _span*

but I am not getting any result. Here's the original search I used as starting point:

| mstats avg(_value) prestats=true WHERE metric_name="kube.container.cpu.usage" AND index="metrics" AND "pod-name"="router*" $mstats_span$ BY "pod-name" | timechart avg(_value) $timechart_span$ agg=max limit=5 useother=false BY "pod-name" | fields - _span*

Re: Using mstats with eval grouped by field

to4kawa — Thu, 23 Jul 2020 10:48:43 GMT

... BY "pod-name" _time

how about adding _time?

Re: Using mstats with eval grouped by field

prandelicious — Thu, 23 Jul 2020 10:53:44 GMT

I modified the search

| mstats avg(kube.pod.cpu.limit) AS cpu_limit avg(kube.pod.cpu.usage_rate) AS cpu_usage WHERE index="metrics" AND "pod-name"="router*" BY "pod-name" span=auto | eval utilization=((cpu_usage/cpu_limit) * 100) | timechart values(utilization) span=5m agg=max limit=5 useother=false BY "pod-name" _time

but I'm getting Error in 'timechart' command: The argument '_time' is invalid.

Re: Using mstats with eval grouped by field

to4kawa — Thu, 23 Jul 2020 11:00:28 GMT

I'm sorry , mstats can't use _time in by clause.

what's this result? is there utilization field ?

Re: Using mstats with eval grouped by field

prandelicious — Thu, 23 Jul 2020 11:30:49 GMT

Hi,

There's no utilization field. Here's the output

Re: Using mstats with eval grouped by field

to4kawa — Thu, 23 Jul 2020 11:49:28 GMT

cpu_limit is null. your eval can't work.
is there kube.pod.cpu.limit ?

Re: Using mstats with eval grouped by field

prandelicious — Thu, 23 Jul 2020 12:01:25 GMT

Yes, there is kube.pod.cpu.limit if I don't use BY pod-name

| mstats avg(kube.pod.cpu.limit) AS "avg_cpu_limit" WHERE index="metrics" span=5m

but I get no result if use BY pod-name

Re: Using mstats with eval grouped by field

to4kawa — Thu, 23 Jul 2020 12:41:47 GMT

| mstats avg(_value) prestats=true WHERE metric_name="kube.pod.cpu.usage_rate" AND index="metrics" AND "pod-name"="router*" span=auto BY "pod-name" | timechart avg(_value) span=auto agg=max limit=5 useother=false BY "pod-name" | fields - _span* | untable _time pod_name usage_avg | append [| mstats avg(_value) prestats=true WHERE metric_name="kube.container.cpu.limit" AND index="metrics" AND "pod-name"="router*" span=auto BY "pod-name" | timechart avg(_value) $timechart_span$ agg=max limit=5 useother=false BY "pod-name" | fields - _span* | untable _time pod_name limit_avg] | stats values(*_avg) as *_avg by _time pod_name

For now, I've combined a working query.
how about this?

Re: Using mstats with eval grouped by field

prandelicious — Fri, 24 Jul 2020 02:31:12 GMT

Thanks for taking time to create the query! Here's the output when I tested it:
what I wanted was to get the value of utilization, which is defined as ((kube.pod.cpu.usage_rate / kube.container.cpu.limit) * 100) that's why I used eval in my previous example.

Re: Using mstats with eval grouped by field

to4kawa — Fri, 24 Jul 2020 03:41:47 GMT

subsearch doesn't return values.

please do troubleshooting.

e.g. subsearch only run, and fix