Splunk Search

Using mstats with eval grouped by field

prandelicious
Loves-to-Learn Lots

I wanted to graph the computed value of two fields and group the result by another field:

 

| mstats avg(kube.pod.cpu.limit) AS cpu_limit avg(kube.pod.cpu.usage_rate) AS cpu_usage WHERE index="metrics" span=auto BY "pod-name"
| eval utilization=((cpu_usage/cpu_limit) * 100)
| timechart values(utilization) agg=max limit=5 useother=false BY "pod-name"
| fields - _span*

 

but I am not getting any result.  Here's the original search I used as starting point:

 

| mstats avg(_value) prestats=true WHERE metric_name="kube.container.cpu.usage" AND index="metrics" AND "pod-name"="router*" $mstats_span$ BY "pod-name"
| timechart avg(_value) $timechart_span$ agg=max limit=5 useother=false BY "pod-name"
| fields - _span*

 

 

 

Labels (3)
0 Karma

to4kawa
Ultra Champion

... BY "pod-name" _time

how about adding _time?

0 Karma

prandelicious
Loves-to-Learn Lots

I modified the search

 

| mstats avg(kube.pod.cpu.limit) AS cpu_limit avg(kube.pod.cpu.usage_rate) AS cpu_usage WHERE index="metrics" AND "pod-name"="router*" BY "pod-name" span=auto
| eval utilization=((cpu_usage/cpu_limit) * 100)
| timechart values(utilization) span=5m agg=max limit=5 useother=false BY "pod-name" _time

 

but I'm getting Error in 'timechart' command: The argument '_time' is invalid.

0 Karma

to4kawa
Ultra Champion

I'm sorry , mstats can't use _time in by clause.

| mstats avg(kube.pod.cpu.limit) AS cpu_limit avg(kube.pod.cpu.usage_rate) AS cpu_usage WHERE index="metrics" AND "pod-name"="router*" BY "pod-name" span=auto
| eval utilization=((cpu_usage/cpu_limit) * 100)

what's this result? is there utilization field ?

0 Karma

prandelicious
Loves-to-Learn Lots

Hi,

There's no utilization field. Here's the output
Screen Shot 2020-07-23 at 7.27.32 PM.png

0 Karma

to4kawa
Ultra Champion

cpu_limit is null. your eval can't work.
is there kube.pod.cpu.limit ?

0 Karma

prandelicious
Loves-to-Learn Lots

Yes, there is kube.pod.cpu.limit if I don't use BY pod-name

| mstats avg(kube.pod.cpu.limit) AS "avg_cpu_limit" WHERE index="metrics" span=5m


Screen Shot 2020-07-23 at 7.56.50 PM.png

but I get no result if use BY pod-name

0 Karma

to4kawa
Ultra Champion
| mstats avg(_value) prestats=true WHERE metric_name="kube.pod.cpu.usage_rate" AND index="metrics" AND "pod-name"="router*" span=auto BY "pod-name"
| timechart avg(_value) span=auto agg=max limit=5 useother=false BY "pod-name"
| fields - _span*
| untable _time pod_name usage_avg
| append [| mstats avg(_value) prestats=true WHERE metric_name="kube.container.cpu.limit" AND index="metrics" AND "pod-name"="router*" span=auto BY "pod-name"
| timechart avg(_value) $timechart_span$ agg=max limit=5 useother=false BY "pod-name"
| fields - _span*
| untable _time pod_name limit_avg]
| stats values(*_avg) as *_avg by _time pod_name 


For now, I've combined a working query.
how about this?

0 Karma

prandelicious
Loves-to-Learn Lots

Thanks for taking time to create the query! Here's the output when I tested it:
Screen Shot 2020-07-24 at 10.21.25 AM.pngwhat I wanted was to get the value of utilization, which is defined as ((kube.pod.cpu.usage_rate / kube.container.cpu.limit) * 100) that's why I used eval in my previous example.

0 Karma

to4kawa
Ultra Champion

subsearch doesn't return values.

please do troubleshooting.

e.g. subsearch only run, and fix

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Can’t Make It to Boston? Stream .conf25 and Learn with Haya Husain

Boston may be buzzing this September with Splunk University and .conf25, but you don’t have to pack a bag to ...

Splunk Lantern’s Guide to The Most Popular .conf25 Sessions

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Unlock What’s Next: The Splunk Cloud Platform at .conf25

In just a few days, Boston will be buzzing as the Splunk team and thousands of community members come together ...