Splunk AppDynamics

Controller only gets part of the data from machine agent, most metrics are missing

Lynn_Chen
Engager

Hi Guys,

I have installed machine agent on a linux server, and connected the agent to one on-premise controller (build 23.4.0-10019) via HTTP . The server appears in controller dashboard correctly, with data in some tables, such as properties, top 10 processes,.., etc. But metrics like Load average, CPU, memory, availability, network, disk usage are always empty.

I checked machine-agent.log, there are ERRORs followed by exception like below:

[system-thread-0] 26 Jul 2023 13:51:47,320 INFO SystemAgent - Using Agent Version [Machine Agent v23.1.0.3555 GA compatible with 4.4.1.0 Build Date 2023-01-24 11:02:32]
...
...
[AD Thread-Metric Reporter1] 26 Jul 2023 14:39:14,547 WARN ManagedMonitorDelegate - Problem registering metrics: Fatal transport error while connecting to URL [/controller/instance/1679/metricregistration]
[AD Thread-Metric Reporter1] 26 Jul 2023 14:39:14,547 WARN ManagedMonitorDelegate - Invalid metric registration response from Controller
[AD Thread-Metric Reporter1] 26 Jul 2023 14:39:14,547 WARN ManagedMonitorDelegate - Problem registering metrics with controller :
java.lang.NullPointerException: null
[AD Thread-Metric Reporter1] 26 Jul 2023 14:39:14,547 ERROR ManagedMonitorDelegate - Error registering metrics
com.singularity.ee.agent.commonservices.metricgeneration.metrics.MetricRegistrationException: Error registering metrics with controller. Response:null
at com.singularity.ee.agent.commonservices.metricgeneration.AMetricSubscriber.doRegisterMetrics(AMetricSubscriber.java:308) ~[shared-22.10.0.34287.jar:?]
at com.singularity.ee.agent.commonservices.metricgeneration.AMetricSubscriber.registerMetrics(AMetricSubscriber.java:156) ~[shared-22.10.0.34287.jar:?]
at com.singularity.ee.agent.commonservices.metricgeneration.MetricGenerationService.registerMetrics(MetricGenerationService.java:329) ~[shared-22.10.0.34287.jar:?]
at com.singularity.ee.agent.commonservices.metricgeneration.MetricReporter.run(MetricReporter.java:101) [shared-22.10.0.34287.jar:?]
at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledExecutorServiceImpl$SafeRunnable.run(AgentScheduledExecutorServiceImpl.java:122) [agent-22.12.0-173.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_362]
at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask$Sync.innerRunAndReset(ADFutureTask.java:335) [agent-22.12.0-173.jar:?]
at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask.runAndReset(ADFutureTask.java:152) [agent-22.12.0-173.jar:?]
at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.access$101(ADScheduledThreadPoolExecutor.java:119) [agent-22.12.0-173.jar:?]
at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.runPeriodic(ADScheduledThreadPoolExecutor.java:206) [agent-22.12.0-173.jar:?]
at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.run(ADScheduledThreadPoolExecutor.java:236) [agent-22.12.0-173.jar:?]
at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.runTask(ADThreadPoolExecutor.java:694) [agent-22.12.0-173.jar:?]
at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.run(ADThreadPoolExecutor.java:726) [agent-22.12.0-173.jar:?]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_362]
Caused by: java.lang.NullPointerException

Is this why some metrics are missing in controller? What might be the reason that only the metricregistration URI timeout but other data is sent to controller correctly?  Any clue is greatly appreciated. 

Labels (1)
0 Karma

rganapavarapu
Path Finder

Hello @Lynn.Chen ,

I would encourage you to open a ticket with debug logs enabled so that one of our MA Support SMEs can help you further. It is hard to comment on the warning message here.

How do I submit a Support ticket? An FAQ 

If you do submit a ticket and hear back from Support, could you please share any learnings back here as a reply? 

Best Regards,
Rajesh Ganapavarapu

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Synthetic Monitoring - Resolved Incident on Detector Alerts

We’ve discovered a bug that affected the auto-clear of Synthetic Detectors in the Splunk Synthetic Monitoring ...

Video | Tom’s Smartness Journey Continues

Remember Splunk Community member Tom Kopchak? If you caught the first episode of our Smartness interview ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud? Learn how unique features like ...