AppD Archive

How to filter FlexClient.FlexClient.pollWithWait from list of problem Business Transactions?

CommunityUser
Splunk Employee
Splunk Employee

Our FlexClient uses the FlexClient.FlexClient.pollWithWait, which can wait for a long time.. not consuming CPU.

These waits are actually fine with us. I am wondering if there is anyway to filter these messages from the bad requests list, so I see only the slow requests that are actually spending excesive time doing work. For example, maybe a regex on the toString() of the stack so I could exclude */flex.messaging.client.FlexClient.pollWithWait/java.lang.Object.wait

from the alerts, or filter on thread name since flex seems to rename the thread when it is in this state.

 

Here is an example "stalled"  request from the List Console.

 

Thanks,

 

-Charles

 

Thread Name:http-8443-5-in-poll-wait
ID:88
Time:Thu Jul 18 15:47:42 PDT 2013
State:TIMED_WAITING
Priority:5

java.lang.Object.wait(Native Method)
flex.messaging.client.FlexClient.pollWithWait(FlexClient.java:861)
flex.messaging.endpoints.BasePollingHTTPEndpoint.handleFlexClientPoll(BasePollingHTTPEndpoint.java:516)
flex.messaging.endpoints.AbstractEndpoint.handleFlexClientPollCommand(AbstractEndpoint.java:1024)
flex.messaging.endpoints.AbstractEndpoint.serviceMessage(AbstractEndpoint.java:844)
flex.messaging.endpoints.amf.MessageBrokerFilter.invoke(MessageBrokerFilter.java:121)
flex.messaging.endpoints.amf.LegacyFilter.invoke(LegacyFilter.java:158)
flex.messaging.endpoints.amf.SessionFilter.invoke(SessionFilter.java:44)
flex.messaging.endpoints.amf.BatchProcessFilter.invoke(BatchProcessFilter.java:67)
flex.messaging.endpoints.amf.SerializationFilter.invoke(SerializationFilter.java:146)
flex.messaging.endpoints.BaseHTTPEndpoint.service(BaseHTTPEndpoint.java:278)
flex.messaging.MessageBrokerServlet.service(MessageBrokerServlet.java:322)
javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
com.vx.servicepeer.sso.SingleSignOnFilter.doFilter(SingleSignOnFilter.java:94)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
com.vx.servicepeer.HttpActivityFilter.doFilter(HttpActivityFilter.java:46)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
com.vx.util.server.ResponseTimeLoggingFilter.doFilter(ResponseTimeLoggingFilter.java:141)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
com.vx.gwtutil.server.Log4jMdcFilter.doFilter(Log4jMdcFilter.java:38)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525)
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:567)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)
java.lang.Thread.run(Thread.java:680)

0 Karma

Arun_Dasetty
Super Champion

Hi Charles,

Thanks for writing to appsphere community forum. We understood that you want to exclude API /flex.messaging.client.FlexClient.pollWithWait/java.lang.Object.wait with the assumption that BT execution does not include

the exec time spent on execluded api, you can exclude packages from Configure -> Call graph settings -> Add custom package to exclude, But this does not help in reducing the BT exec time. as AD monitors transaction but does not affect the execution of any API on server.

Please let us know inreasing stall thresholds to value higher than total BT exec time under Configure -> Thresholds -> Stall threshold (default 45 secs) helps? , so that you will see  snapshots with call graph symbol

refer docs: http://litedocs.appdynamics.com/display/ADLite/Configure+Thresholds

Let us know if that answers your query, Do write back to us for further assistance.

Thanks,

Arun

0 Karma

CommunityUser
Splunk Employee
Splunk Employee

Arun -

Increasing the stall times is not helpful because I do need to know about the request that are slow doing actual work.

I just want to filter out the ones that are being marked as stalls, but are actually TIMED_WAITS, which is a normal condition on our server.

Is this sort of filtering I am asking about available in the commercial product?

Thanks.

Charles

0 Karma

Arun_Dasetty
Super Champion

Hi Charles,

Thanks for the update. In Pro version of product we have feature to define exclude rules, in which you can define class and method names or based on url, this helps in discovering flex.messaging.client.FlexClient.pollWithWait as a business transaction, let us know if that meets your requirement

refer docs or refer attachment if you have trouble accessing doc: http://docs.appdynamics.com/display/PRO13S/Configure+Business+Transaction+Detection#ConfigureBusines... let us know if that helps

Thanks,

Arun

0 Karma

CommunityUser
Splunk Employee
Splunk Employee

Thanks Arun --

 

The doc you attached does not offer a solution, since the filters are all on the request.

I dont know that the transaction is one that can be ignored until the call gets all the way into the

 

java.lang.Object.wait(Native Method)

 

and only if it is called from 

 

flex.messaging.client.FlexClient.pollWithWait(FlexClient.java:861)

 

at which point we are _many_ calls beyond the entry point where the filter is applied.

 

This is why I was hoping to apply a filter to the entire call stack, not to the entry point and its parameters.

 

This does not seem to be expressible in the filtering language in the doc you attached.

My login does not give me access to the full documentation set, so I cannot determine if this could be acheived in some other way.

 

-Charles

0 Karma

Arun_Dasetty
Super Champion

Hi Charles,

Thanks for posting back.  Currently we do not see any option apart from the following mentioned:

- Servelt1 -> calls -> classA:methodA calls -> ClassB:methodB -> calls -> flex.messaging.client.FlexClient.pollWithWait and does spent time in wait method and then calls -> ClassC:methodC

and say  we see Servlet1 as BT here, we do not see any option apart from execluding flex.messaging.client.FlexClient.pollWithWait API in exclude rules which helps to avoid monitoring as a BT, but this does not seems to help in your case as you are seeing the flex API as part of different BT/entry points.

 

We apologize for the incovenience, as we could not see any option apart from defining an exclude rule on FlexClient.pollWithWait API under Configure -> instrumentation -> transaction detection -> exclude rules section

Let us know if you have any further concerns

Thanks,

Arun

0 Karma
Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...