Jboss EAP 6.1- transaction snapshots not getting captured

[CommunityUser]

We executed a long running test on our server having APPDYNAMICS  configured.

Though we are seeing lot of stats but transaction snapshots are empty.They are not showing any info whether it is error ones,slow ones etc..

I have seen earlier such data getting captured when I was using Jboss 5...

I have configured Jboss 6 as ebelow:

added "com.singularity" to the below line

set "JAVA_OPTS=%JAVA_OPTS% -Djboss.modules.system.pkgs=org.jboss.byteman,com.singularity"
rem adding appdynamics
set JAVA_OPTS=%JAVA_OPTS% -javaagent:C:/AppServerAgent/javaagent.jar=uniqueID=abced.

In agent logs I can see 

RequestSegmentDataQueue - Request Segment Data Reporter Queue full. Dropping snapshot.

Have I missed anything here ??

One thing I would like to tell you is our log configruration had messed up and log files are more than 10 GB now .I hope Appdynamics doesnt collect data from logs.

Also I would like to know how transaction snapshot is picked up by appdyanmics.

Please let me know incase you need any more info.

[Arun_Dasetty]

Hi,

We understood business trasnactions are discovered for the agent node monitoring the jboss eap 6.x under the corresponding tier name in <associated application> -> Business trasnaction screen in controller UI, and the discovered BTs have stats but found no snapshots, let us know if it is otherwise

From the error message provided "RequestSegmentDataQueue - Request Segment Data Reporter Queue full. Dropping snapshot." it seems like snapshot limit reached and this could be due too many snapshots in a minute and we might be hitting max-snapshots-per-minute limit which is default 15, but we still expect some snapshots to be reported

Can you please send the logs under <agent_install_dir>/logs path to assist you better?

Also provide screenshots from controller UI depicting the issue?

Regards,

Arun

[CommunityUser]

Thanks Arun for replying...Looks like our infrastructure team restarted the jboss due to to excessive logging issue.After jboss restart I am able to see transaction snapshots being captured.

Has transaction snapshots anything to do with jboss logs ?

Also I was analysing some older saved performance runs,while they have all the stats and graphs stored but transaction snapshots are missing in older ones..I can assure you that they were present before...

I would appreciate if you can answer these queries.

[Arun_Dasetty]

Hi,

Glad to hear that news from you. Please find the response to your queries:

>>>>>>. Has transaction snapshots anything to do with jboss logs ?

Answer:   No, we do not expect jboss logging to affect snapshot detection, but we suspect that either agent have network connnectivity issues or snapshot limit had hit during the problem and restarting/resetting agent will reset the limit counters which might have helped here.

>>>>.. Also I was analysing some older saved performance runs,while they have all the stats and graphs stored but transaction snapshots are missing in older ones..I can assure you that they were present before...?

Answer: By default, snapshots reported and persisted in controller database will be purged/deleted from controller system after two weeks as stated in docs: http://docs.appdynamics.com/display/PRO13S/Database+Size%2C+Data+Retention%2C+and+Metric+Resolution, However you can increase the limit but we do not recommend that as it takes more disk space and add overhead on controller performance.

Let us know if that information helps.

Regards,

Arun

[CommunityUser]

Thanks for letting me know the rule for storage of transaction snapshots..

Also can you please let me know when and why I am getting below messages:

Reporter Queue full. Dropping metrics.

Why is queue getting full ?

What does this mean?

Can I increase the queue size ?

Thanks,

Rips

[Arun_Dasetty]

Hi,

Please find the response to your queries inline:

>>>>> Also can you please let me know when and why I am getting below messages:
Reporter Queue full. Dropping metrics. ?

Answer: Following are the scenarios that the error raises:

- agent failed to connect controller due to network or someother issue and after sometime agent internal buffer fills up and reaches the queue lenght and logs the same in the agent logs

- there are too many snapshots or metrics in internal queue and had reached queue limit than you will see such error

>>>> Why is queue getting full ?

Answer: Due to internal limits imposed by agent to store metrics/snapshots and other data to make sure agent is not adding memory overhead when there the metircs/snapshots limit is reached

>>>> What does this mean?

Answer: It means the number of metrics/snapshots that can be added to the queue is reached its max limit.

>>>> Can I increase the queue size ?

Answer: you cannot increse the queue size directly as the error you provided raised for different limits, but can you try

increasing snapshots limit and metric limit as such:

max-snapshots-per-min=15 (default value, for the node level property )

max-concurrent-snapshots=500 (default value, for the node level property )
-Dappdynamics.agent.maxMetrics=6000 (jvm arg)

Let us know if that answers your query.

Regards,

Arun

[CommunityUser]

Hi Arun,

Though earlier after jboss restart snapshots started getting captured but now again I am seeing below msg in Appagent log:

 Request Segment Data Reporter Queue full. Dropping snapshot.

 I dont think this is a network issue as we are able to receive other metrics but only snapshots are missing.

Snapshots are not being captured at all right now.

Thanks ,

Rips

[Arun_Dasetty]

Hi,

Can you please send the zipped version of agent logs from the agent node for which there are no snapshots collected which  helps us assist you better?

agent logs path can be found under <agent_install_dir>/logs directory

Regards,

Arun

[CommunityUser]

I cleaned up the folders under conf and logs created with the name 'unique node' and restared the server.

Since then it is working smoothly.

If it happens again I will keep the logs and attach here.

[Arun_Dasetty]

Hi,

Glad to hear that news that the issue is currently not exists, we request you to revisti the snapshot threshold setting and periodic/occurance snapshot settings to make sure there are optimized , having too aggressive thresholds and occurance sampling might lead to reach limits frequently., you can set the settings under Configure -> slow transaction thresholds

If the issue happens again send the zipped version of agent we will be glad to assist you.

Regards,

Arun

[CommunityUser]

I dont think we have agressive thresholds and I dont think that is the reason for snapshots not getting captured...

After cleaning the appgents conf and log folder,it has been running smoothly.Do I need to do this before everytime I run a long running test ?

[Arun_Dasetty]

Hi,

Agent have internal limit and will get reset on agent restart / reset and we request you to do agent reset (this does not require restart and helps in resetting agent limits back) as referred in below screenshot:

And if you still see the issue we request you to send the agent logs which helps to debug the issue to find the root cause.

Regards,

Arun

[CommunityUser]

Thanks for this info..I will let you know in case I find the issue again.

[CommunityUser]

Unfortunately We are again facing the issue.PFA logs

[Arun_Dasetty]

Hi,

We apologize for the incovenience here, We see the following error in agent logs which is evident there are issues in reporting snapshots from agent end:

======================

[AD Thread Pool-Global1] 19 Feb 2014 06:55:45,232  WARN AgentErrorProcessor - Agent error occurred, [name,transformId]=[com.singularity.snapshot.RequestSegmentDataQueue - javax.xml.parsers.FactoryConfigurationError,2147483647]
[AD Thread Pool-Global1] 19 Feb 2014 06:55:45,232  WARN AgentErrorProcessor - 3 instance(s) remaining before error log is silenced
[AD Thread Pool-Global1] 19 Feb 2014 06:55:45,232 ERROR RequestSegmentDataQueue - Error in processing snapshot
javax.xml.parsers.FactoryConfigurationError: Provider __redirected.__DocumentBuilderFactory could not be instantiated: java.lang.NullPointerException
    at javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:129)

==============================

We have access to your saas Controller and we could see that threshold settings are fine and looks optimized.

Can you please file a ticket at https://help.appdynamics.com/tickets/new or you can send email to help@appdynamics.com with zipped version of agent logs in debug mode (you can change the logging level from "info" to "debug" for only 10 minutes in log4j.xml file under path C:\AppServerAgent\conf\logging ) during load on BTs and attach the logs under path <agent_install_dir>/logs

We will file internal bug once you filed a ticket for better tracking purpose.

Regards,

Arun