Splunk Search

Using mstats with eval grouped by field

prandelicious
Loves-to-Learn Lots

I wanted to graph the computed value of two fields and group the result by another field:

 

| mstats avg(kube.pod.cpu.limit) AS cpu_limit avg(kube.pod.cpu.usage_rate) AS cpu_usage WHERE index="metrics" span=auto BY "pod-name"
| eval utilization=((cpu_usage/cpu_limit) * 100)
| timechart values(utilization) agg=max limit=5 useother=false BY "pod-name"
| fields - _span*

 

but I am not getting any result.  Here's the original search I used as starting point:

 

| mstats avg(_value) prestats=true WHERE metric_name="kube.container.cpu.usage" AND index="metrics" AND "pod-name"="router*" $mstats_span$ BY "pod-name"
| timechart avg(_value) $timechart_span$ agg=max limit=5 useother=false BY "pod-name"
| fields - _span*

 

 

 

Labels (3)
0 Karma

to4kawa
Ultra Champion

... BY "pod-name" _time

how about adding _time?

0 Karma

prandelicious
Loves-to-Learn Lots

I modified the search

 

| mstats avg(kube.pod.cpu.limit) AS cpu_limit avg(kube.pod.cpu.usage_rate) AS cpu_usage WHERE index="metrics" AND "pod-name"="router*" BY "pod-name" span=auto
| eval utilization=((cpu_usage/cpu_limit) * 100)
| timechart values(utilization) span=5m agg=max limit=5 useother=false BY "pod-name" _time

 

but I'm getting Error in 'timechart' command: The argument '_time' is invalid.

0 Karma

to4kawa
Ultra Champion

I'm sorry , mstats can't use _time in by clause.

| mstats avg(kube.pod.cpu.limit) AS cpu_limit avg(kube.pod.cpu.usage_rate) AS cpu_usage WHERE index="metrics" AND "pod-name"="router*" BY "pod-name" span=auto
| eval utilization=((cpu_usage/cpu_limit) * 100)

what's this result? is there utilization field ?

0 Karma

prandelicious
Loves-to-Learn Lots

Hi,

There's no utilization field. Here's the output
Screen Shot 2020-07-23 at 7.27.32 PM.png

0 Karma

to4kawa
Ultra Champion

cpu_limit is null. your eval can't work.
is there kube.pod.cpu.limit ?

0 Karma

prandelicious
Loves-to-Learn Lots

Yes, there is kube.pod.cpu.limit if I don't use BY pod-name

| mstats avg(kube.pod.cpu.limit) AS "avg_cpu_limit" WHERE index="metrics" span=5m


Screen Shot 2020-07-23 at 7.56.50 PM.png

but I get no result if use BY pod-name

0 Karma

to4kawa
Ultra Champion
| mstats avg(_value) prestats=true WHERE metric_name="kube.pod.cpu.usage_rate" AND index="metrics" AND "pod-name"="router*" span=auto BY "pod-name"
| timechart avg(_value) span=auto agg=max limit=5 useother=false BY "pod-name"
| fields - _span*
| untable _time pod_name usage_avg
| append [| mstats avg(_value) prestats=true WHERE metric_name="kube.container.cpu.limit" AND index="metrics" AND "pod-name"="router*" span=auto BY "pod-name"
| timechart avg(_value) $timechart_span$ agg=max limit=5 useother=false BY "pod-name"
| fields - _span*
| untable _time pod_name limit_avg]
| stats values(*_avg) as *_avg by _time pod_name 


For now, I've combined a working query.
how about this?

0 Karma

prandelicious
Loves-to-Learn Lots

Thanks for taking time to create the query! Here's the output when I tested it:
Screen Shot 2020-07-24 at 10.21.25 AM.pngwhat I wanted was to get the value of utilization, which is defined as ((kube.pod.cpu.usage_rate / kube.container.cpu.limit) * 100) that's why I used eval in my previous example.

0 Karma

to4kawa
Ultra Champion

subsearch doesn't return values.

please do troubleshooting.

e.g. subsearch only run, and fix

0 Karma
Get Updates on the Splunk Community!

Index This | Why did the turkey cross the road?

November 2025 Edition  Hayyy Splunk Education Enthusiasts and the Eternally Curious!   We’re back with this ...

Enter the Agentic Era with Splunk AI Assistant for SPL 1.4

  🚀 Your data just got a serious AI upgrade — are you ready? Say hello to the Agentic Era with the ...

Feel the Splunk Love: Real Stories from Real Customers

Hello Splunk Community,    What’s the best part of hearing how our customers use Splunk? Easy: the positive ...