All Apps and Add-ons
Highlighted

CPU Load on AIX - getting different values from TA-nmon (events), TA-metricator-for-nmon (metrics) for the same host

Path Finder

Hello all,

Two Nmon TAs on one AIX host.
TA-nmon (based on events) showing CPU load higher than TA-metricator-for-nmon (based on metrics).

I need help to figure out what might cause this.

SPL on data from TA-nmon (events):

| tstats 
avg(CPU.UPTIME.load_average_1min) AS load_average_1min 
avg(CPU.UPTIME.load_average_5min) AS load_average_5min 
avg(CPU.UPTIME.load_average_15min) AS load_average_15min 
from datamodel=NMON_Data_CPU 
where (nodename = CPU.UPTIME) host= 
groupby _time host prestats=true `nmon_span`
| timechart `nmon_span` 
avg(CPU.UPTIME.load_average_1min) AS load_average_1min 
avg(CPU.UPTIME.load_average_5min) AS load_average_5min 
avg(CPU.UPTIME.load_average_15min) AS load_average_15min

SPL on data from TA-metricator-for-nmon (metrics):

| mstats avg(_value) as value where `nmon_metrics_index` 
(metric_name=os.unix.nmon.system.uptime.load_average_1min OR 
metric_name=os.unix.nmon.system.uptime.load_average_5min OR 
metric_name=os.unix.nmon.system.uptime.load_average_15min) 
host= by metric_name `nmon_span` 
| `extract_metrics("load_average_1min load_average_5min load_average_15min")`
| fillnull value=0 load_average_1min load_average_5min load_average_15min 
| timechart `nmon_span` 
avg(load_average_1min) as load_average_1min 
avg(load_average_5min) as load_average_5min 
avg(load_average_15min) as load_average_15min

alt text

Update:
Continuing my investigation.
Source is output from "uptime" command which is exactly the same for both events and metrics TAs.

From
/opt/splunkforwarder/etc/apps/(TA-metricator-for-nmon OR TA-nmon)/bin/nmonexternalcmd/nmonexternalsnap.sh

# Uptime information (uptime command output)
echo "UPTIME,$1,\"`uptime | sed 's/^\s//g' | sed 's/,/;/g'`\"" >>NMON_FIFO_PATH/nmon_external.dat &

Turns out that data based on events is accurate, but metric values are exactly 3 times less (attached screenshot).
alt text

0 Karma
Highlighted

Re: CPU Load on AIX - getting different values from TA-nmon (events), TA-metricator-for-nmon (metrics) for the same host

Ultra Champion

SPL on data from TA-nmon (events):

add span in tstats

0 Karma
Highlighted

Re: CPU Load on AIX - getting different values from TA-nmon (events), TA-metricator-for-nmon (metrics) for the same host

Path Finder

Hi @to4kawa
Thanks for the suggestion, my aim is to migrate from event based dashboards to metrics based, that's why I am currently checking if metrics are providing exactly the same data as events.

0 Karma
Highlighted

Re: CPU Load on AIX - getting different values from TA-nmon (events), TA-metricator-for-nmon (metrics) for the same host

Path Finder

Hey Alexey,
Remove live 7 in your SPL

fillnull value=0 loadaverage1min loadaverage5min loadaverage15min

The fillnull is splitting it into 3 events and then the timechart is generating the average across these ( as two of them have a 0 value it is essentially dividing by 3).

View solution in original post

0 Karma
Highlighted

Re: CPU Load on AIX - getting different values from TA-nmon (events), TA-metricator-for-nmon (metrics) for the same host

Path Finder

Indeed, this fixed the problem. Thank you very much, Mel !
So, Metricator for NMON -> "UPTIME Load Average" panel's SPL is to be slightly corrected, working version:

| mstats avg(_value) as value where `nmon_metrics_index` 
 (metric_name=os.unix.nmon.system.uptime.load_average_1min OR 
 metric_name=os.unix.nmon.system.uptime.load_average_5min OR 
 metric_name=os.unix.nmon.system.uptime.load_average_15min) 
 host= by metric_name `nmon_span` 
 | `extract_metrics("load_average_1min load_average_5min load_average_15min")`
 | timechart `nmon_span` 
 avg(load_average_1min) as load_average_1min 
 avg(load_average_5min) as load_average_5min 
 avg(load_average_15min) as load_average_15min
0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.