All Apps and Add-ons

On SCK from my Kubernetes metrics, can I calculate the memory percentage usage per node ?

yannK
Splunk Employee
Splunk Employee

I have SCK setup, and collect my Kubernetes metrics. We have access out of the box to the node memory limit kube.node.memory.allocatable (in MB), and to the memory usage kube.node.memory.working_set_bytes (in bytes)

but we want to do some calculations to get the memory usage percentage per node.

Labels (1)
Tags (1)
0 Karma
1 Solution

yannK
Splunk Employee
Splunk Employee

If you use the Graphs in the analytics UI are based on those searches

 

| mstats avg("kube.node.memory.allocatable") prestats=true WHERE "index"="k8s_metrics" span=10s
| timechart avg("kube.node.memory.allocatable") AS Avg span=10s
| fields - _span*

 

and

 

 

| mstats avg("kube.node.memory.working_set_bytes") prestats=true WHERE "index"="k8s_metrics" span=10s
| timechart avg("kube.node.memory.working_set_bytes") AS Avg span=10s
| fields - _span*

 

 

To extract them on a single search and do the calculation, you can use that method.

 

| mstats avg("kube.node.memory.allocatable") As alloc_MB avg("kube.node.memory.working_set_bytes") AS working_B prestat=f WHERE "index"="k8s_metrics" span=10s by node
| timechart avg(eval(100*working_B/(alloc_MB*1024*1024))) AS mem_used_percent span=10s by node

 

 

 

Key points : need to use AS to cast the results into fields you can manipulate later. Use an EVAL in the function to do the calculation on the fly. For timecharts ensure that the span in the mstats is equal or smaller than the span in the timechart. Use a split BY the appropriate field, here by node. We also disabled the "prestat" from the mstats, otherwise the fields names  would be slightly different.

View solution in original post

0 Karma

yannK
Splunk Employee
Splunk Employee

If you use the Graphs in the analytics UI are based on those searches

 

| mstats avg("kube.node.memory.allocatable") prestats=true WHERE "index"="k8s_metrics" span=10s
| timechart avg("kube.node.memory.allocatable") AS Avg span=10s
| fields - _span*

 

and

 

 

| mstats avg("kube.node.memory.working_set_bytes") prestats=true WHERE "index"="k8s_metrics" span=10s
| timechart avg("kube.node.memory.working_set_bytes") AS Avg span=10s
| fields - _span*

 

 

To extract them on a single search and do the calculation, you can use that method.

 

| mstats avg("kube.node.memory.allocatable") As alloc_MB avg("kube.node.memory.working_set_bytes") AS working_B prestat=f WHERE "index"="k8s_metrics" span=10s by node
| timechart avg(eval(100*working_B/(alloc_MB*1024*1024))) AS mem_used_percent span=10s by node

 

 

 

Key points : need to use AS to cast the results into fields you can manipulate later. Use an EVAL in the function to do the calculation on the fly. For timecharts ensure that the span in the mstats is equal or smaller than the span in the timechart. Use a split BY the appropriate field, here by node. We also disabled the "prestat" from the mstats, otherwise the fields names  would be slightly different.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...