AppD Archive

Unable to view custom JMX metrics

CommunityUser
Splunk Employee
Splunk Employee

Dear,

We have running JBoss application and want to view some HornetQ queues parameters in the JMX metrics.

The HronetQ JMX metrics are enabled in the configuration.

In the JMX metrics there is only the option to view the JVM and JDBC connection pools.

Kind regards,

Joakim

0 Karma
1 Solution

Arun_Dasetty
Super Champion

Hi Joakim,

Can you try the following steps and let us know how it goes:

- Make sure agent system clock is in sync with controller time and do agent restart

- Do an agent reset and create a metric on object pattern without quotes as in old rule mbean pattern, refer below screenshot and see whether it goes fine with test metric:

image.png

Above said time skew issues and agent connectivity errors are evident from below logs:

[AD Thread Pool-Global90] 04 Dec 2014 06:32:11,508 WARN ConfigurationChannel - Invalid response for configuration request from controller/could not connect to controller. Msg : Fatal transport error: Connection reset..
[AD Thread Pool-Global101] 04 Dec 2014 11:37:46,484 INFO ControllerTimeSkewHandler - Agent timestamp: 1417689466484; Controller timestamp: 1417689520988; Skew value=54504 msec; agentLagging=true

- If the test metrc discovered fine, Can you delete the new rule we created by wrong submit under JMX config and add "name"  without quotes as value for field insance identifier in screen referred in screenshot JMX%20metric%20rules.png and save the changes and do an "agent reset" node dashboard -> agents screen and see how it goes

Regards,

Arun

View solution in original post

0 Karma

Arun_Dasetty
Super Champion

Hi Joakim,

Can you click "Mbean browser" in the same screenshot provided to us and check what Hornetmq mbean categories and and object pattern of the mbeans exposed are in sync with JMX  config under Configure -> Instrumentation -> JMX screen you are referring ? 

If the rule and object pattenr of mbeans exposed of target mbeans in "mbean browser" are not in sycn , you would need to update either jmx rule existing as per object pattern of mbeans exposed in mbean browser Or create rules from mbean browser itself, Let us know if that information helps.

Regards,

Arun

0 Karma

CommunityUser
Splunk Employee
Splunk Employee

Hi,

The object pattern is in sync with the Mbean in the JMX MBean browser.

I have one rule created directly from the MBean browser called 'S-Up Queueu' Messages and the other one is a default appdynamics rule. Both object patterns are correct conform the Mbean browser. See screenshot.

Kind regards,

Joakim

0 Karma

Arun_Dasetty
Super Champion

Hi Joakim,

Can you export jmx config xml for config have both the rules from Configure -> JMX  -> select config having the two ruled configued (default and custom) and click "export" option in controller UI

And also  provide screenshot from node dashboard -> JMX -> MBean browser and provided screenshots for the mbean in view with same object pattern in mbean browser in addition to agent logs from affected agent node where you provide screenshot from mbean browser from node dashboard to assist you better.

Regards,

Arun

0 Karma

CommunityUser
Splunk Employee
Splunk Employee

Hi Arun,

I attached logs, Mbean screenshot and the org.hornetq metric rules.

Kind regards,

Joakim

0 Karma

Arun_Dasetty
Super Champion

Hi Joakim,

Can you try the following steps and let us know how it goes:

- Make sure agent system clock is in sync with controller time and do agent restart

- Do an agent reset and create a metric on object pattern without quotes as in old rule mbean pattern, refer below screenshot and see whether it goes fine with test metric:

image.png

Above said time skew issues and agent connectivity errors are evident from below logs:

[AD Thread Pool-Global90] 04 Dec 2014 06:32:11,508 WARN ConfigurationChannel - Invalid response for configuration request from controller/could not connect to controller. Msg : Fatal transport error: Connection reset..
[AD Thread Pool-Global101] 04 Dec 2014 11:37:46,484 INFO ControllerTimeSkewHandler - Agent timestamp: 1417689466484; Controller timestamp: 1417689520988; Skew value=54504 msec; agentLagging=true

- If the test metrc discovered fine, Can you delete the new rule we created by wrong submit under JMX config and add "name"  without quotes as value for field insance identifier in screen referred in screenshot JMX%20metric%20rules.png and save the changes and do an "agent reset" node dashboard -> agents screen and see how it goes

Regards,

Arun

0 Karma

CommunityUser
Splunk Employee
Splunk Employee

Hi,

Well it seems that the skew was the factor in the metrics failure.

Object patterns with quotes are also showing up in the metrics browser.

Thank you!

Kr,

Joakim

0 Karma
Get Updates on the Splunk Community!

Fueling your curiosity with new Splunk ILT and eLearning courses

At Splunk Education, we’re driven by curiosity—both ours and yours! That’s why we’re committed to delivering ...

Splunk AI Assistant for SPL 1.1.0 | Now Personalized to Your Environment for Greater ...

Splunk AI Assistant for SPL has transformed how users interact with Splunk, making it easier than ever to ...

Unleash Unified Security and Observability with Splunk Cloud Platform

     Now Available on Microsoft AzureOn Demand Now Step boldly into the AI revolution with enhanced security ...