Splunk AppDynamics

App Agent Not Reporting

CommunityUser
Splunk Employee
Splunk Employee

Hi Team,

We've an application configured in PROD region recently. When i was about to check the health by today evening it was not reporting data for last 2 hrs.

So i tried to check APp agent status in Tier and Node UI, it shows 0 % and i restarted APP'D agent coordinator process in the App server.

But its not coming up still, could someone pls help how can i get reporting back.

Regards,
Soundarajan

Labels (1)
0 Karma

CommunityUser
Splunk Employee
Splunk Employee

Your log shows a lot of exceptions relating to Network communication, check the exceptions under your application on the controller, you will see most of these listed in their as well.

Check the network connectivity between the application server and the controller on the designated port.

I strongly advise you to approach support if the issue persists.

st.GetRequestStream()
at AppDynamics.Shared.Communication.sg.qs(Byte[] , String )
at com.appdynamics.ee.rest.controller.request.sd.wap()]
2018-04-01 19:59:01.7800 6448 w3wp 3 11 Warn AgentErrorProcessor Agent error occurred, [name,transformId]=[com.appdynamics.CONFIG.ConfigurationChannel - System.Net.WebException,11]
2018-04-01 19:59:01.7800 6448 w3wp 3 11 Warn AgentErrorProcessor 4 instance(s) remaining before error log is silenced
2018-04-01 19:59:01.7800 6448 w3wp 3 11 Warn AgentErrorProcessor 499 instance(s) remaining before instrumentation point is targeted for neutralization
2018-04-01 19:59:01.7800 6448 w3wp 3 11 Error ConfigurationChannel Exception: The operation has timed out Exception: System.Net.WebException: The operation has timed out
at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
at System.Net.HttpWebRequest.GetRequestStream()
at AppDynamics.Shared.Communication.sg.qs(Byte[] , String )
at com.appdynamics.ee.rest.controller.request.sd.wap()

0 Karma

Abhilash_C
Explorer

Hi Madan,

I am facing a kind of similar but strange issue where app agent for Few of nodes alone not getting Reported although sufficient load is put. Please see the Below Logs. I am able to see Node Property is getting auto disabled. Can you help me with this ?image.png

0 Karma

CommunityUser
Splunk Employee
Splunk Employee

@Abhilash.C Interesting set of log entries, I have seen similar behaviour when you have more than one node agent with the same name reporting under the same application, especially with the combination of server visibility + .NET agent and machine agent messing around.

I have too little context on the infra you are runiing on or the history of the issue, and no access to full log details[Please don't post the logs here I am aginst people doing it]

I would say delete your logs and rung the agent for a couple of minutes and then copy the log file and read through every line, this will probably give better context of whats happenning. Appdynamics support used to be great in dealing with issues of this sort, good luch getting your problem sorted by the @support team

0 Karma

Raunak_Mohanty
Builder

Hi,

  We will need to figure out why Agent stopped reporting suddenly. Were there any planned activities executed around the time like server restart or AppPool recycle?

After restarting Coordinator service have you tried restarting IIS/Instrumented application as well? If not please try and verify if Agents come back up

Thanks,

Raunak

0 Karma

CommunityUser
Splunk Employee
Splunk Employee

Hi Raunak,

No, there was not any activities happened. It stopped reporting suddenly.

But we've restarted the agent service in that server and still its not reporting data back.

Also unable to restart IIS due to Production environment.

Is IIS restart only option to solve this ?

Could you please advise on this. Thanks

Regards,

Soundarajan

0 Karma

CommunityUser
Splunk Employee
Splunk Employee
agentlog is your only friend when you are looking to answer a question of "Why"
0 Karma

CommunityUser
Splunk Employee
Splunk Employee

Hi Madan,

Ya i tried investigating agent logs, but i don't see much information about the APP Agent issue. I've attached here for reference.

Regards,
Soundarajan

[ @Anonymous - Please do not attach logs or any possiblity sensitive materials that may be compromising security and privacy concerns. Thank you, @Jeanie.Kedia. ]

0 Karma

CommunityUser
Splunk Employee
Splunk Employee
Mate,

DO NOT post your agentlogs on community site. Delete these if possible.

CommunityUser
Splunk Employee
Splunk Employee

Whenever you instrument an IIS application you will have to go through the below checklist. Please check if you did these in the exact same sequence.

1. Install Agent
2. Configure Agent
3. Restart Agent co-ordinator service
4. Reset IIS
5. Simulate traffic and make sure the tier, node, and the simulated load are captured in appdynamimcs.

0 Karma

CommunityUser
Splunk Employee
Splunk Employee

Hi Madan,

Ya i tried investigating agent logs, but i don't see much information about the APP Agent issue. I've attached here for reference.

Regards,
Soundarajan

@Anonymous - Please do not attach logs or any possiblity sensitive materials that may be compromising security and privacy concerns. Thank you, @Jeanie.Kedia. ]

0 Karma
Get Updates on the Splunk Community!

Technical Workshop Series: Splunk Data Management and SPL2 | Register here!

Hey, Splunk Community! Ready to take your data management skills to the next level? Join us for a 3-part ...

Splunk Observability Synthetic Monitoring - Resolved Incident on Detector Alerts

We’ve discovered a bug that affected the auto-clear of Synthetic Detectors in the Splunk Synthetic Monitoring ...

Video | Tom’s Smartness Journey Continues

Remember Splunk Community member Tom Kopchak? If you caught the first episode of our Smartness interview ...