Splunk Search

Using mstats with eval grouped by field

prandelicious
Loves-to-Learn Lots

I wanted to graph the computed value of two fields and group the result by another field:

 

| mstats avg(kube.pod.cpu.limit) AS cpu_limit avg(kube.pod.cpu.usage_rate) AS cpu_usage WHERE index="metrics" span=auto BY "pod-name"
| eval utilization=((cpu_usage/cpu_limit) * 100)
| timechart values(utilization) agg=max limit=5 useother=false BY "pod-name"
| fields - _span*

 

but I am not getting any result.  Here's the original search I used as starting point:

 

| mstats avg(_value) prestats=true WHERE metric_name="kube.container.cpu.usage" AND index="metrics" AND "pod-name"="router*" $mstats_span$ BY "pod-name"
| timechart avg(_value) $timechart_span$ agg=max limit=5 useother=false BY "pod-name"
| fields - _span*

 

 

 

Labels (3)
0 Karma

to4kawa
Ultra Champion

... BY "pod-name" _time

how about adding _time?

0 Karma

prandelicious
Loves-to-Learn Lots

I modified the search

 

| mstats avg(kube.pod.cpu.limit) AS cpu_limit avg(kube.pod.cpu.usage_rate) AS cpu_usage WHERE index="metrics" AND "pod-name"="router*" BY "pod-name" span=auto
| eval utilization=((cpu_usage/cpu_limit) * 100)
| timechart values(utilization) span=5m agg=max limit=5 useother=false BY "pod-name" _time

 

but I'm getting Error in 'timechart' command: The argument '_time' is invalid.

0 Karma

to4kawa
Ultra Champion

I'm sorry , mstats can't use _time in by clause.

| mstats avg(kube.pod.cpu.limit) AS cpu_limit avg(kube.pod.cpu.usage_rate) AS cpu_usage WHERE index="metrics" AND "pod-name"="router*" BY "pod-name" span=auto
| eval utilization=((cpu_usage/cpu_limit) * 100)

what's this result? is there utilization field ?

0 Karma

prandelicious
Loves-to-Learn Lots

Hi,

There's no utilization field. Here's the output
Screen Shot 2020-07-23 at 7.27.32 PM.png

0 Karma

to4kawa
Ultra Champion

cpu_limit is null. your eval can't work.
is there kube.pod.cpu.limit ?

0 Karma

prandelicious
Loves-to-Learn Lots

Yes, there is kube.pod.cpu.limit if I don't use BY pod-name

| mstats avg(kube.pod.cpu.limit) AS "avg_cpu_limit" WHERE index="metrics" span=5m


Screen Shot 2020-07-23 at 7.56.50 PM.png

but I get no result if use BY pod-name

0 Karma

to4kawa
Ultra Champion
| mstats avg(_value) prestats=true WHERE metric_name="kube.pod.cpu.usage_rate" AND index="metrics" AND "pod-name"="router*" span=auto BY "pod-name"
| timechart avg(_value) span=auto agg=max limit=5 useother=false BY "pod-name"
| fields - _span*
| untable _time pod_name usage_avg
| append [| mstats avg(_value) prestats=true WHERE metric_name="kube.container.cpu.limit" AND index="metrics" AND "pod-name"="router*" span=auto BY "pod-name"
| timechart avg(_value) $timechart_span$ agg=max limit=5 useother=false BY "pod-name"
| fields - _span*
| untable _time pod_name limit_avg]
| stats values(*_avg) as *_avg by _time pod_name 


For now, I've combined a working query.
how about this?

0 Karma

prandelicious
Loves-to-Learn Lots

Thanks for taking time to create the query! Here's the output when I tested it:
Screen Shot 2020-07-24 at 10.21.25 AM.pngwhat I wanted was to get the value of utilization, which is defined as ((kube.pod.cpu.usage_rate / kube.container.cpu.limit) * 100) that's why I used eval in my previous example.

0 Karma

to4kawa
Ultra Champion

subsearch doesn't return values.

please do troubleshooting.

e.g. subsearch only run, and fix

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

[Puzzles] Solve, Learn, Repeat: Character substitutions with Regular Expressions

This challenge was first posted on Slack #puzzles channelFor BORE at .conf23, we had a puzzle question which ...

Splunk Community Badges!

  Hey everyone! Ready to earn some serious bragging rights in the community? Along with our existing badges ...

[Puzzles] Solve, Learn, Repeat: Matching cron expressions

This puzzle (first published here) is based on matching timestamps to cron expressions.All the timestamps ...