Splunk Search

Using mstats with eval grouped by field

prandelicious
Loves-to-Learn Lots

I wanted to graph the computed value of two fields and group the result by another field:

 

| mstats avg(kube.pod.cpu.limit) AS cpu_limit avg(kube.pod.cpu.usage_rate) AS cpu_usage WHERE index="metrics" span=auto BY "pod-name"
| eval utilization=((cpu_usage/cpu_limit) * 100)
| timechart values(utilization) agg=max limit=5 useother=false BY "pod-name"
| fields - _span*

 

but I am not getting any result.  Here's the original search I used as starting point:

 

| mstats avg(_value) prestats=true WHERE metric_name="kube.container.cpu.usage" AND index="metrics" AND "pod-name"="router*" $mstats_span$ BY "pod-name"
| timechart avg(_value) $timechart_span$ agg=max limit=5 useother=false BY "pod-name"
| fields - _span*

 

 

 

Labels (3)
0 Karma

to4kawa
Ultra Champion

... BY "pod-name" _time

how about adding _time?

0 Karma

prandelicious
Loves-to-Learn Lots

I modified the search

 

| mstats avg(kube.pod.cpu.limit) AS cpu_limit avg(kube.pod.cpu.usage_rate) AS cpu_usage WHERE index="metrics" AND "pod-name"="router*" BY "pod-name" span=auto
| eval utilization=((cpu_usage/cpu_limit) * 100)
| timechart values(utilization) span=5m agg=max limit=5 useother=false BY "pod-name" _time

 

but I'm getting Error in 'timechart' command: The argument '_time' is invalid.

0 Karma

to4kawa
Ultra Champion

I'm sorry , mstats can't use _time in by clause.

| mstats avg(kube.pod.cpu.limit) AS cpu_limit avg(kube.pod.cpu.usage_rate) AS cpu_usage WHERE index="metrics" AND "pod-name"="router*" BY "pod-name" span=auto
| eval utilization=((cpu_usage/cpu_limit) * 100)

what's this result? is there utilization field ?

0 Karma

prandelicious
Loves-to-Learn Lots

Hi,

There's no utilization field. Here's the output
Screen Shot 2020-07-23 at 7.27.32 PM.png

0 Karma

to4kawa
Ultra Champion

cpu_limit is null. your eval can't work.
is there kube.pod.cpu.limit ?

0 Karma

prandelicious
Loves-to-Learn Lots

Yes, there is kube.pod.cpu.limit if I don't use BY pod-name

| mstats avg(kube.pod.cpu.limit) AS "avg_cpu_limit" WHERE index="metrics" span=5m


Screen Shot 2020-07-23 at 7.56.50 PM.png

but I get no result if use BY pod-name

0 Karma

to4kawa
Ultra Champion
| mstats avg(_value) prestats=true WHERE metric_name="kube.pod.cpu.usage_rate" AND index="metrics" AND "pod-name"="router*" span=auto BY "pod-name"
| timechart avg(_value) span=auto agg=max limit=5 useother=false BY "pod-name"
| fields - _span*
| untable _time pod_name usage_avg
| append [| mstats avg(_value) prestats=true WHERE metric_name="kube.container.cpu.limit" AND index="metrics" AND "pod-name"="router*" span=auto BY "pod-name"
| timechart avg(_value) $timechart_span$ agg=max limit=5 useother=false BY "pod-name"
| fields - _span*
| untable _time pod_name limit_avg]
| stats values(*_avg) as *_avg by _time pod_name 


For now, I've combined a working query.
how about this?

0 Karma

prandelicious
Loves-to-Learn Lots

Thanks for taking time to create the query! Here's the output when I tested it:
Screen Shot 2020-07-24 at 10.21.25 AM.pngwhat I wanted was to get the value of utilization, which is defined as ((kube.pod.cpu.usage_rate / kube.container.cpu.limit) * 100) that's why I used eval in my previous example.

0 Karma

to4kawa
Ultra Champion

subsearch doesn't return values.

please do troubleshooting.

e.g. subsearch only run, and fix

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Take Action Automatically on Splunk Alerts with Red Hat Ansible Automation Platform

 Are you ready to revolutionize your IT operations? As digital transformation accelerates, the demand for ...

Calling All Security Pros: Ready to Race Through Boston?

Hey Splunkers, .conf25 is heading to Boston and we’re kicking things off with something bold, competitive, and ...

Beyond Detection: How Splunk and Cisco Integrated Security Platforms Transform ...

Financial services organizations face an impossible equation: maintain 99.9% uptime for mission-critical ...