Splunk Search

Using mstats with eval grouped by field

prandelicious
Loves-to-Learn Lots

I wanted to graph the computed value of two fields and group the result by another field:

 

| mstats avg(kube.pod.cpu.limit) AS cpu_limit avg(kube.pod.cpu.usage_rate) AS cpu_usage WHERE index="metrics" span=auto BY "pod-name"
| eval utilization=((cpu_usage/cpu_limit) * 100)
| timechart values(utilization) agg=max limit=5 useother=false BY "pod-name"
| fields - _span*

 

but I am not getting any result.  Here's the original search I used as starting point:

 

| mstats avg(_value) prestats=true WHERE metric_name="kube.container.cpu.usage" AND index="metrics" AND "pod-name"="router*" $mstats_span$ BY "pod-name"
| timechart avg(_value) $timechart_span$ agg=max limit=5 useother=false BY "pod-name"
| fields - _span*

 

 

 

Labels (3)
0 Karma

to4kawa
Ultra Champion

... BY "pod-name" _time

how about adding _time?

0 Karma

prandelicious
Loves-to-Learn Lots

I modified the search

 

| mstats avg(kube.pod.cpu.limit) AS cpu_limit avg(kube.pod.cpu.usage_rate) AS cpu_usage WHERE index="metrics" AND "pod-name"="router*" BY "pod-name" span=auto
| eval utilization=((cpu_usage/cpu_limit) * 100)
| timechart values(utilization) span=5m agg=max limit=5 useother=false BY "pod-name" _time

 

but I'm getting Error in 'timechart' command: The argument '_time' is invalid.

0 Karma

to4kawa
Ultra Champion

I'm sorry , mstats can't use _time in by clause.

| mstats avg(kube.pod.cpu.limit) AS cpu_limit avg(kube.pod.cpu.usage_rate) AS cpu_usage WHERE index="metrics" AND "pod-name"="router*" BY "pod-name" span=auto
| eval utilization=((cpu_usage/cpu_limit) * 100)

what's this result? is there utilization field ?

0 Karma

prandelicious
Loves-to-Learn Lots

Hi,

There's no utilization field. Here's the output
Screen Shot 2020-07-23 at 7.27.32 PM.png

0 Karma

to4kawa
Ultra Champion

cpu_limit is null. your eval can't work.
is there kube.pod.cpu.limit ?

0 Karma

prandelicious
Loves-to-Learn Lots

Yes, there is kube.pod.cpu.limit if I don't use BY pod-name

| mstats avg(kube.pod.cpu.limit) AS "avg_cpu_limit" WHERE index="metrics" span=5m


Screen Shot 2020-07-23 at 7.56.50 PM.png

but I get no result if use BY pod-name

0 Karma

to4kawa
Ultra Champion
| mstats avg(_value) prestats=true WHERE metric_name="kube.pod.cpu.usage_rate" AND index="metrics" AND "pod-name"="router*" span=auto BY "pod-name"
| timechart avg(_value) span=auto agg=max limit=5 useother=false BY "pod-name"
| fields - _span*
| untable _time pod_name usage_avg
| append [| mstats avg(_value) prestats=true WHERE metric_name="kube.container.cpu.limit" AND index="metrics" AND "pod-name"="router*" span=auto BY "pod-name"
| timechart avg(_value) $timechart_span$ agg=max limit=5 useother=false BY "pod-name"
| fields - _span*
| untable _time pod_name limit_avg]
| stats values(*_avg) as *_avg by _time pod_name 


For now, I've combined a working query.
how about this?

0 Karma

prandelicious
Loves-to-Learn Lots

Thanks for taking time to create the query! Here's the output when I tested it:
Screen Shot 2020-07-24 at 10.21.25 AM.pngwhat I wanted was to get the value of utilization, which is defined as ((kube.pod.cpu.usage_rate / kube.container.cpu.limit) * 100) that's why I used eval in my previous example.

0 Karma

to4kawa
Ultra Champion

subsearch doesn't return values.

please do troubleshooting.

e.g. subsearch only run, and fix

0 Karma
Get Updates on the Splunk Community!

The OpenTelemetry Certified Associate (OTCA) Exam

What’s this OTCA exam? The Linux Foundation offers the OpenTelemetry Certified Associate (OTCA) credential to ...

From Manual to Agentic: Level Up Your SOC at Cisco Live

Welcome to the Era of the Agentic SOC   Are you tired of being a manual alert responder? The security ...

Splunk Classroom Chronicles: Training Tales and Testimonials (Episode 4)

Welcome back to Splunk Classroom Chronicles, our ongoing series where we shine a light on what really happens ...