AppD Archive

Custom Metrics not appearing

CommunityUser
Splunk Employee
Splunk Employee

Im in the process of creating a few cuistom metrics with AppDynamices.

I have done the following

1. Created the a new directory under monitors

2. Created the monitor.xml with 775 permissions

3. Created shell script with 775 permissions and tested from command line.

4. Restarted Machine Agent.

The issue I'm finding, is that its not showing up any place within the control (maybe I not looking in correct place). I can see its running via the log file

[Agent-Monitor-Scheduler-3] 20 Jan 2017 15:08:14,568  INFO PeriodicTaskRunner - Returning time out value of [60000] ms for monitor task [Run]
[Agent-Monitor-Scheduler-3] 20 Jan 2017 15:08:14,568  INFO ExecTask - Running Executable Command [[/opt/appdynamics/machine-agent/monitors/slpUsage/userUsage.sh]]
[Agent-Monitor-Scheduler-3] 20 Jan 2017 15:08:14,569  INFO ExecTask - Started Executable Command [[/opt/appdynamics/machine-agent/monitors/slpUsage/userUsage.sh]]
[Agent-Monitor-Scheduler-3] 20 Jan 2017 15:08:15,612  INFO MonitorStreamConsumer - Stopping monitored process

Here is the monitor.xml

<monitor>
    <name>UserUsage</name>
    <type>managed</type>
    <description>Monitor open file count </description>
    <monitor-configuration></monitor-configuration>
    <monitor-run-task>
        <execution-style>periodic</execution-style>
        <execution-frequency-in-seconds>60</execution-frequency-in-seconds>
        <execution-timeout-in-secs>60</execution-timeout-in-secs>
        <name>Run</name>
        <type>executable</type>
        <task-arguments></task-arguments>
        <executable-task>
            <type>file</type>
            <file>userUsage.sh</file>
        </executable-task>
    </monitor-run-task>
</monitor>

0 Karma

CommunityUser
Splunk Employee
Splunk Employee

Where your custom metrics show in the metric browser is dependent on how your userUsage.sh is echoing its data to standard output.  A good place to start looking for your metrics is in 

Application Infrastructure Performance

|_Tier

    |_Individual Nodes

      |_<node name>

        |_Custom Metrics

If you don't see the metrics there, then enable logging in <MACHINE_AGENT_HOME>/conf/logging/log4j.xml and restart the machine agent.  THen review your machine-agent.loc file to see if your script is erroring out.

Another good tip is to simply run your shell script on the command line.  If you have structured the output properly and there are no floating point integers then it should roll into the controller under the structure that your script outputs.

0 Karma

CommunityUser
Splunk Employee
Splunk Employee

 Does the metric need to be named with Custom Metrics or can it be named anything.

Reason I ask, is I only see metrics showing up when named Custom Metrics

name=Custom Metrics|Users|xxxxxx01|Active,aggregator=OBSERVATION,value=57
name=Custom Metrics|Users|xxxxxx02|Active,aggregator=OBSERVATION,value=45
name=Stats|Users|xxxx-abc-01|Active,aggregator=OBSERVATION,value=155
name=Stats|Users|xxxx-abc-02|Active,aggregator=OBSERVATION,value=151

The Custom Metrics show up, but Stats does not

0 Karma

CommunityUser
Splunk Employee
Splunk Employee

Here's an example stdout from a custom monitor i've deployed on 100's of VM's.  

name=Custom Metrics|Hardware Resources|CPU|Total CPUs, value=2
name=Custom Metrics|Hardware Resources|CPU|Load Average, value=1
name=Custom Metrics|Hardware Resources|CPU|Load Util%, value=29
name=Custom Metrics|Hardware Resources|Memory|Swap Total(MB), value=8443
name=Custom Metrics|Hardware Resources|Memory|Swap Used(MB), value=174
name=Custom Metrics|Hardware Resources|Memory|Swap Free(MB), value=8268
name=Custom Metrics|Hardware Resources|Memory|Swap Used%, value=2
name=Custom Metrics|Database Resources|Database|Database Status, value=1
name=Custom Metrics|Database Resources|Listener|Listener Status, value=1

To me, your stdout looks like there are some values that don't need to be in there...namely the ",aggregator=OBSERVATION" portion of your output.  If you enable debug on your machine agent and tail the log file you should see some error messages when the agent tries to run your custom monitor.

0 Karma

CommunityUser
Splunk Employee
Splunk Employee
They need to roll up under custom metrics...in my experience
0 Karma
Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...