AppD Archive

Custom Metrics not visible in the UI/Metric Browser

CommunityUser
Splunk Employee
Splunk Employee

I've written a simple script to output a line of data with a value, following the cutom metrics and machine agent docs pages.

This is a simple shell script.

However there's some conflicting info in those docs as to whether a full specification of aggregator, roll-up etc. parameters are required in the output.

In any case, i've tried both per example for Custom Metrics and also for existing position in the metrics tree and I'm still unable to get this output visible anywhere in the metrics tree. Yes I've looked under app infra performance.

Any help would be appreciated.

0 Karma

Arun_Dasetty
Super Champion

Hi Jason,

Can you please provide logs under <MachineAgent_dir>/logs path and attach script file creating the custom metric and also provide screenshot from metric browser with "Hardware resources" expanded we usually see custom metrics in parallel to hardware resources section., to assist you further please provide above logs.

Regards,

Arun

0 Karma

CommunityUser
Splunk Employee
Splunk Employee

 

Machine Log extract:

[Agent-Monitor-Scheduler-1] 06 Apr 2015 23:17:16,204 INFO PeriodicTaskRunner - Periodic Task - setup metric feed for [Run]
[Agent-Monitor-Scheduler-1] 06 Apr 2015 23:17:16,204 INFO PeriodicTaskRunner - Returning time out value of [120000] ms for monitor task [Run]
[Agent-Monitor-Scheduler-1] 06 Apr 2015 23:17:16,204 INFO ExecTask - Running Executable Command [[/sbclocal/apps/ad/monitors/ad_memcheck/memcheck2.sh]]
[Agent-Monitor-Scheduler-1] 06 Apr 2015 23:17:16,211 INFO ExecTask - Started Executable Command [[/sbclocal/apps/ad/monitors/ad_memcheck/memcheck2.sh]]
[Agent-Monitor-Scheduler-1] 06 Apr 2015 23:17:17,235 INFO MonitorStreamConsumer - Stopping monitored process

 

Script outputs:

name=Custom Metrics|Hardware Resources|Unused_Mb,value=1778,aggregator=OBSERVATION,time-rollup=CURRENT,cluster-rollup=INDIVIDUAL

 

I've also tried:

name=Server|Component:179|Hardware Resources|Memory|Unused Mb,value=1862,aggregator=OBSERVATION,time-rollup=CURRENT,cluster-rollup=INDIVIDUAL

 

 - where component 179 is a custom tier I created.

 

Script is:

#!/bin/bash

unused()
{
s=/proc/meminfo
let unused=$(grep ^MemFree $s|awk '{T+=$2}END{print T}')/1024
let memfree=$(egrep 'MemFree|Buffers|^Cached' $s|awk '{T+=$2}END{print T}')/1024
let swfree=$(sed -n /^SwapFree:/p $s|awk '{print $2}')/1024
let totfree=($swfree+$memfree)
echo "name=Custom Metrics|Hardware Resources|Unused_Mb,value=$unused,aggregator=OBSERVATION,time-rollup=CURRENT,cluster-rollup=INDIVIDUAL"
}
unused

Nothing I try works, and I've bounced the machine agent between each change.

0 Karma

Arun_Dasetty
Super Champion

Hi Jason, 

Please provide the machine agent logs folder archive for our review

0 Karma

CommunityUser
Splunk Employee
Splunk Employee

I can now see the custom metric tree appearing for this but not the actual values.

Please contact me privately for any more data/logs.

Thanks.

0 Karma
Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...