topic Re: Machine Agent Http Listener not working in Splunk AppDynamics

Machine Agent Http Listener not working

David_Machacek — Sat, 15 Apr 2023 13:50:02 GMT

We have same problem as described in https://community.appdynamics.com/t5/Controller-SaaS-On-Premises/Machine-Agent-Http-Listener-not-working/td-p/46694.

We followed https://docs.appdynamics.com/appd/22.x/22.2/en/infrastructure-visibility/machine-agent/extensions-and-custom-metrics/machine-agent-http-listener, we send metrics, we get 204. But no data are displayed in the GUI console. Only the new metrics are registered, but no values are shown.

We send the data from synthetics jobs using python script. Logs from my script:

[INFO] Request sent, body=[{"metricName":"Custom Metrics|WebVitals|LCP","aggregatorType":"AVERAGE","value":9695.514}], responseStatus=204

Re: Machine Agent Http Listener not working

Sunil_Agarwal — Sat, 15 Apr 2023 18:05:03 GMT

Hi @David.Machacek , Is this issue occurring on a SAAS or on-premises controller? If it is on-premises, you can review the controller logs to determine if there was a problem with how the controller handled the Custom Metrics. In order to acquire more information to troubleshoot this problem, you might also need to activate debug logging for that machine agent.

Re: Machine Agent Http Listener not working

David_Machacek — Sat, 15 Apr 2023 19:08:17 GMT

It is SAAS controller. But I check the machine agent logs where we find errors:

Apr 15, 2023 6:56:15 PM com.sun.jersey.spi.container.ContainerResponse mapMappableContainerException
SEVERE: The RuntimeException could not be mapped to a response, re-throwing to the HTTP container
java.lang.NullPointerException
at com.appdynamics.sim.agent.extensions.rest.AgentResource.writeMetrics(AgentResource.java:37)
at jdk.internal.reflect.GeneratedMethodAccessor159.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)

What are this caused by? We follow the documentation in our request (Http POST to machine agent in our OpenShift cluster/service) with body:
[{"metricName":"Custom Metrics|WebVitals|LCP","aggregatorType":"AVERAGE","value":9695.514}]

Re: Machine Agent Http Listener not working

Morelz — Sun, 16 Apr 2023 10:46:14 GMT

Hi David

For your custom Metrics , please try changing your metrics by adding "value" at the end

For example, your metric field is

Custom Metrics|WebVitals|LCP

Try using

Custom Metrics|WebVitals|LCP|value

We did extensive work in creating the custom metrics and decided on using this structure. This also will assist you when you want to dashboard a large number of metrics, where you can use a wildcard to show multiple values on the same graph, instead of adding multiple data points per metric. AppD doesn't support wildcarding the last name in the metric tree, hence adding the value at the end.

Also push some load on the endpoint initially, we also had a case where we needed to push some more load than a couple of requests, and after that it worked normally

Ciao

Re: Machine Agent Http Listener not working

David_Machacek — Sun, 16 Apr 2023 13:29:56 GMT

@Mario.Morelli thanks for your reply. I did try that but it did not help. We still do not see any metrics in GUI.

[INFO] Request sent, body=[{"metricName":"Custom Metrics|WebVitals|LCP|value","aggregatorType":"AVERAGE","value":7823.73}], responseStatus=204

I have sent around 100 of requests. But at least we dont have any more error logs in machine agent. Do you have any other suggestion?

Re: Machine Agent Http Listener not working

Morelz — Mon, 17 Apr 2023 08:50:02 GMT

Hey David

Can you do the following for the final test before troubleshooting

Send a couple of requests every 2-3 minutes and change the "number values" you provide for every request. Just remember, AppDynamics stores data in intervals, I believe the minimum values for metrics are 1 minute. So it can only store 1 value per minute, it's not on a millisecond level.

Provide feedback around the above if the graph starts displaying:)

Ciao

Re: Machine Agent Http Listener not working

David_Machacek — Mon, 17 Apr 2023 09:42:37 GMT

We continue testing. But since we use this feature to collect additional metrics for synthetic testing, which cannot more frequently then once in 5 minute (limitation given by synthetic monitoring jobs). But ill use some curl-based script to try it out more often.

Also currently we suspect, the floating points in our values might be to blame. AppDynamics might expect only whole numbers.

I will give update in couple of hours.

Re: Machine Agent Http Listener not working

David_Machacek — Mon, 17 Apr 2023 10:25:32 GMT

So I ran following script for an hour, but without any success.

while $true; do curl -XPOST 'http://ma-agent.namespace.svc.cluster.local:8293/api/v1/metrics' -H 'Content-Type: application/json;charset=UTF-8' --data-raw '[{"metricName":"Custom Metrics|WebVitals|David|value","aggregatorType":"AVERAGE","value":'"$((RANDOM % 10 + 1))"'}]' -k -v ; sleep 1; done

I still dont know what is wrong. Can you help us please?

Re: Machine Agent Http Listener not working

Morelz — Tue, 18 Apr 2023 06:32:26 GMT

Hey David

I used your example, just manually tested and it works, I am just running locally with port 8082 instead of default port but doesn't make a difference

curl -v -X POST 'http://localhost:8082/api/v1/metrics' -H 'Content-Type: application/json;charset=UTF-8' --data-raw '[{"metricName":"Custom Metrics|WebVitals|David|value","aggregatorType":"AVERAGE","value":104}]'

And it works

Just make sure the value is not in single quotes, integer values should not be enclosed in single quotes. Think that might be your problem

Ciao

Re: Machine Agent Http Listener not working

Morelz — Tue, 18 Apr 2023 06:48:31 GMT

Hi David

I have tested myself with single and double quotes and both seem to work, so doesn't appear to be that.

Can you test 5 values to replicate my testing, sending one vlaue every 1-2 minutes

50,80, 104, 500, 800

Also just to confirm the following, do you have a SIM License enabled for the machine agent?

The only other reason you would get this behavior of the metric name registered but the values not is due to that server where the http listener is running not having a SIM license and value in controller-info.xml for sim-enabled is not set to true

Ciao

Re: Machine Agent Http Listener not working

David_Machacek — Mon, 24 Apr 2023 12:03:13 GMT

Hello Mario. Thanks for help. The problem was indeed in SIM license enabling in AppD configuration. Thanks!

Re: Machine Agent Http Listener not working

David_Machacek — Mon, 24 Apr 2023 12:25:14 GMT

Thanks, problem was in SIM lincesing. APPDYNAMICS_SIM_ENABLED were to false. Thanks