How to understand java.lang.Object:wait in group of snapshots

[Kenny_Yu]

The method java.lang.Object:wait shows prominently under "Most Expensive Methods" from "Identify the most expensive calls from a group of snapshots", together with "Spring Bean:process" and MessageContext:send.

None of those methods show on the individual snapshot's "Call Drill Down". I figure they were filtered out.

Under "Hot Spots", I figure only the methods with high self-time are listed.

As a result, the hotspots don't agree with the most expensive calls in a group at all. Since I can't trace up from the most expensive calls, I don't find them useful.

Since I can't find the expensive method java.lang.Object:wait in the call graph, nor the hotspot list, I can't figure out what that wait is for.

Any help will be appreciated.

Kenny

PS: AppD v3.8.5.1

[Arun_Dasetty]

Hi Kenny,

Can you provide the screenshots from controller UI depicting the issue and an exported pdf report of the snapshot and screenshot with packages excludes from call graph in UI as shown in below screenshot to debug further:

Regards,

Arun

[Kenny_Yu]

Hi, Arun,

I am attaching screenshots

1. Expensive methods in a group of snapshot. The group, however, contains only one snapshot.

2. The same snapshot's hotspots

3. The package excluded list

I'd expect 1 and 2 to be in agreement, or corroboration, confirming each other on the hotspots. A simple understanding is that 1 is the aggregate of many 2's. They appear very different.

Thanks for the reply.

Kenny

[Arun_Dasetty]

Hi Kenny,

We could not locate object.wait in call graph and hotspots of other , Can you please remove filter and expand all the stack traces Or you can attach the snapshot using "Export to pdf" button and attach to this post for more clarity?

While you provide screenshot can you please force include the packages that were involved object.wait method call , refer doc section "To Include Specific Sub-packages (Sub-namespaces) or Classes from the Excluded Packages" at https://docs.appdynamics.com/display/PRO40/Configure+Call+Graphs

Try to include packages excluded in screenshot Pkg%20Excludes.png and see if you found more sub calls in call graph for calles to object.wait api?

Regards,

Arun

[Kenny_Yu]

Hi, Arun,

As you requested, I am attaching new docs on the same snapshot:

1. The call graph exported in pdf

2. The hotspot view that seems to be the self-methods taken from the call graph

3. The most expensive calls view from Identify the most expensive calls

Configuring the package exclusion is not possible: java.* is listed as "Required:YES" in the exclusion list. Adding java.lang.* in the inclusion list made no difference - Object:wait doesn't show up.

The confusion is that the Object:wait takes 600 ms in the "expensive calls" list, but i don't have a way of tracing to the invoker, therefore, can't get to the source of it. 

Thanks

Kenny

[Arun_Dasetty]

Hi Kenny,

Can you provide the application name in the saas controller UI ? so that we will check the behavior in UI itself by searching for snapshot GUID for transaction snapshot report provided to us.

We suspect you are trying to view expensive methods in multiple requests as the total response time of snapshot in screenshot ExpensiveMethodsinGroup.png is less than the next expensive method after wait method which is send method

Also let us know how you have navigated to ExpensiveMethodsinGroup.png screen popup, we understood that you have selected view expensive methods option on snapshot context menu but would like get clarity on exact steps used.

Regards,

Arun

[Kenny_Yu]

Hi, Arun,

The app and component names are in the link https://dtv.saas.appdynamics.com/controller/index.html#/location=APP_COMPONENT_MANAGER&timeRange=las...

I'd go to "Transactin Snapshots", then "Slow and Errored Transactins". Many under URL PROJ_ADE_PDR_235450 will show up with Exec TIme >1 sec. 

To get to the Most Expensive view, I right-click on a transaction, and pick "Identify the most expensive...". My intent was to select a few transactions and run on the selected group. But for the purpose of comparing with snapshot view, I only select one such that the only difference is the viewing method.

Again, the confusion is that some calls in the most expensive view (Object:wait, for example) don't show in call graph, nor the hotspot view, therere, I can't trace up to the app code.

Thanks

[Arun_Dasetty]

Hi Kenny,

Somehow the saas UI deep link failed to load at my end, apologies for that, but please confirm if this is the case, When you drill down "All other traffic - <tier>" BT is that a distribute flow view if that is the case, Can you please point the methods that is just higher or equal to the execution of the time taken by Object:wait and see if those method exec time is in sync by any chance?

Regards,

Arun

[Kenny_Yu]

Hi, Arun,

The flow map is very simple - I pick the far back end to start drilling. All calls happen within the JEE container, except a few fast sql calls taking less than 100 ms. 

As the Most Expensive chart shows, two other calls take longer than Object:wait. None of them show up on the HotSpots view. 

I can do a screen sharing with you to clear things up if that helps.

Thanks

Kenny

[Arun_Dasetty]

Hi Kenny,

Can you help us raise help ticket at help ticket at https://help.appdynamics.com/tickets/new Or send email to help@appdynamics.com to speed up the process , please attach the screenshots from UI and deep link of latest snapshot 

On of our Support team will send the meeting details based on there observations from saas UI , i requested to raise ticket as somehow the saas UI is not loading frpm my end where other saas urls are opening fine, Please raise help ticket as suggested so that we could speed up process in addrssing your concern.

Regards,

Arun

[Ashish_Mehta]

Hi Kenny,

Refer the attached screenshots for the same transaction (that spans 3 tiers)

This particular transaction spans across 3 tiers and each individual tier will have its own call graph captured which is linked to each other via same Request GUID.

originating from tier = NGN-OSB
Continuing into tiers = ADE-ADS-VBD and ADE-CxCore

As the originating tier is NGN-OSB, the snapshot picture will show links to all the 3 tiers, but when you look into other individual tiers snapshot parts, it will only show the call graphs specific to its own leg of the transaction.

The most expensive call (via right click) will show the expensive calls in all the legs of the snapshot (call graph) or in other words for all the selected request GUIDs.

Now in your case, when you select a particular snapshot and look for most expensive method calls, it will show the result aggregated from all the 3 tiers' call graphs but the object.wait() calls are part of hot spots (and call graph) of snapshot leg spent in NGN-OSB tier.

Let me know if that makes sense or not or you need more clarification.

Thanks,
Ashish Mehta

[Kenny_Yu]

Ashish,

Your explanation makes sense. So the "Most expensive calls" view is based on a wider scope spanning all tiers. Since the calls are not linked to the corresponding call stack, I have to go up or down the tiers to locate their context. Whereas on the hotspot view, each call is linked to the full stack in the lower panel. Am I missing anything?

Thanks

Kenny

[Ashish_Mehta]

Hi Kenny,

Yes, your understanding is right.

Hot spots is specific to a particular call stack, if you are looking at the call stack of the snapshot in a particular tier then it will shown the expensive calls (host spots) in call graph of that tier only.

Best is to open the snapshot in the orginating tier because that will have links to all downstream tiers as well with drill down into their call stacks (if the calls stack were taken in those tiers)

Thanks,

Ashish Mehta