I have SCK setup, and collect my Kubernetes metrics. We have access out of the box to the node memory limit kube.node.memory.allocatable (in MB), and to the memory usage kube.node.memory.working_set_bytes (in bytes)
but we want to do some calculations to get the memory usage percentage per node.
If you use the Graphs in the analytics UI are based on those searches
| mstats avg("kube.node.memory.allocatable") prestats=true WHERE "index"="k8s_metrics" span=10s
| timechart avg("kube.node.memory.allocatable") AS Avg span=10s
| fields - _span*
and
| mstats avg("kube.node.memory.working_set_bytes") prestats=true WHERE "index"="k8s_metrics" span=10s
| timechart avg("kube.node.memory.working_set_bytes") AS Avg span=10s
| fields - _span*
To extract them on a single search and do the calculation, you can use that method.
| mstats avg("kube.node.memory.allocatable") As alloc_MB avg("kube.node.memory.working_set_bytes") AS working_B prestat=f WHERE "index"="k8s_metrics" span=10s by node
| timechart avg(eval(100*working_B/(alloc_MB*1024*1024))) AS mem_used_percent span=10s by node
Key points : need to use AS to cast the results into fields you can manipulate later. Use an EVAL in the function to do the calculation on the fly. For timecharts ensure that the span in the mstats is equal or smaller than the span in the timechart. Use a split BY the appropriate field, here by node. We also disabled the "prestat" from the mstats, otherwise the fields names would be slightly different.
If you use the Graphs in the analytics UI are based on those searches
| mstats avg("kube.node.memory.allocatable") prestats=true WHERE "index"="k8s_metrics" span=10s
| timechart avg("kube.node.memory.allocatable") AS Avg span=10s
| fields - _span*
and
| mstats avg("kube.node.memory.working_set_bytes") prestats=true WHERE "index"="k8s_metrics" span=10s
| timechart avg("kube.node.memory.working_set_bytes") AS Avg span=10s
| fields - _span*
To extract them on a single search and do the calculation, you can use that method.
| mstats avg("kube.node.memory.allocatable") As alloc_MB avg("kube.node.memory.working_set_bytes") AS working_B prestat=f WHERE "index"="k8s_metrics" span=10s by node
| timechart avg(eval(100*working_B/(alloc_MB*1024*1024))) AS mem_used_percent span=10s by node
Key points : need to use AS to cast the results into fields you can manipulate later. Use an EVAL in the function to do the calculation on the fly. For timecharts ensure that the span in the mstats is equal or smaller than the span in the timechart. Use a split BY the appropriate field, here by node. We also disabled the "prestat" from the mstats, otherwise the fields names would be slightly different.