I'm hoping someone could help.
I would like to create a dashboard for one of our hosts similar to the Splunk app for infrastructure overview page (as in screenshot below).
We are indexing as metrics (index=em_metrics) and would like to create the panels shown below.
I think you are looking for SPLs to use for your dashboard i.e "mstats" command: https://docs.splunk.com/Documentation/Splunk/8.0.0/SearchReference/Mstats
You can use commands like mstats, mcatalog to look into em_metrics index.
Sample SPL for your dashboard:
| mstats avg(memory.free) WHERE index=em_metrics AND host=xyzz span=1m
You can split the above search by any dimension.
To get the list of dimensions:
| mcatalog values(_dims) WHERE index=em_metrics AND host=xyzz
You can pipe it to timechart to produce a chart: https://docs.splunk.com/Documentation/Splunk/8.0.0/SearchReference/Timechart
I have been using the mstats command to create metrics dashboards etc but what I cant seem to figure out is how the CPU is calculated. I cant seem to get the same result. I am not sure what metric_name is used to get the required result CPU used %.
Only metrics are:
You should use Processor.%_Processor_Time for Cpu utilization.
See this to learn more about this metric: https://social.technet.microsoft.com/wiki/contents/articles/12984.understanding-processor-processor-...
Processor - % Processor Time is the percentage of elapsed time that the processor spends to execute a non-Idle thread. It is calculated by measuring the percentage of time that the processor spends executing the idle thread and then subtracting that value from 100%. (Each processor has an idle thread that consumes cycles when no other threads are ready to run). This counter is the primary indicator of processor activity, and displays the average percentage of busy time observed during the sample interval.