All Apps and Add-ons

On SCK from my Kubernetes metrics, can I calculate the memory percentage usage per node ?

yannK
Splunk Employee
Splunk Employee

I have SCK setup, and collect my Kubernetes metrics. We have access out of the box to the node memory limit kube.node.memory.allocatable (in MB), and to the memory usage kube.node.memory.working_set_bytes (in bytes)

but we want to do some calculations to get the memory usage percentage per node.

Labels (1)
Tags (1)
0 Karma
1 Solution

yannK
Splunk Employee
Splunk Employee

If you use the Graphs in the analytics UI are based on those searches

 

| mstats avg("kube.node.memory.allocatable") prestats=true WHERE "index"="k8s_metrics" span=10s
| timechart avg("kube.node.memory.allocatable") AS Avg span=10s
| fields - _span*

 

and

 

 

| mstats avg("kube.node.memory.working_set_bytes") prestats=true WHERE "index"="k8s_metrics" span=10s
| timechart avg("kube.node.memory.working_set_bytes") AS Avg span=10s
| fields - _span*

 

 

To extract them on a single search and do the calculation, you can use that method.

 

| mstats avg("kube.node.memory.allocatable") As alloc_MB avg("kube.node.memory.working_set_bytes") AS working_B prestat=f WHERE "index"="k8s_metrics" span=10s by node
| timechart avg(eval(100*working_B/(alloc_MB*1024*1024))) AS mem_used_percent span=10s by node

 

 

 

Key points : need to use AS to cast the results into fields you can manipulate later. Use an EVAL in the function to do the calculation on the fly. For timecharts ensure that the span in the mstats is equal or smaller than the span in the timechart. Use a split BY the appropriate field, here by node. We also disabled the "prestat" from the mstats, otherwise the fields names  would be slightly different.

View solution in original post

0 Karma

yannK
Splunk Employee
Splunk Employee

If you use the Graphs in the analytics UI are based on those searches

 

| mstats avg("kube.node.memory.allocatable") prestats=true WHERE "index"="k8s_metrics" span=10s
| timechart avg("kube.node.memory.allocatable") AS Avg span=10s
| fields - _span*

 

and

 

 

| mstats avg("kube.node.memory.working_set_bytes") prestats=true WHERE "index"="k8s_metrics" span=10s
| timechart avg("kube.node.memory.working_set_bytes") AS Avg span=10s
| fields - _span*

 

 

To extract them on a single search and do the calculation, you can use that method.

 

| mstats avg("kube.node.memory.allocatable") As alloc_MB avg("kube.node.memory.working_set_bytes") AS working_B prestat=f WHERE "index"="k8s_metrics" span=10s by node
| timechart avg(eval(100*working_B/(alloc_MB*1024*1024))) AS mem_used_percent span=10s by node

 

 

 

Key points : need to use AS to cast the results into fields you can manipulate later. Use an EVAL in the function to do the calculation on the fly. For timecharts ensure that the span in the mstats is equal or smaller than the span in the timechart. Use a split BY the appropriate field, here by node. We also disabled the "prestat" from the mstats, otherwise the fields names  would be slightly different.

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...