Splunk AppDynamics

JVM crash with OpenJDK, Appd Agent and G1GC

CommunityUser
Splunk Employee
Splunk Employee

Hi,

JVM crashes with no information about the crash. Here are the details of our applications and environment :

OpenJDK version : 1.8.0_91

AppD Controller : 4.5.10.8362

GC : G1GC

I have already tried the option mentioned here:

https://community.appdynamics.com/t5/Java-Java-Agent-Installation-JVM/JVM-crash-with-AppD-agent-and-...

But still getting the crashes.

Can some one provide some pointers.

Labels (1)

Peter_Holditch
Builder

Vikas,

1.8.0_91 is extremely old (dating from April 2016) and there are a huge number of bug fixes in the subsequent releases.

Experience shows that JVM crashes of this kind are almost always fixed by upgrading to the latest JVM version, so I would recommend you try that (and make sure to pay attention to Oracle's licensing changes)

Warm regards,

Peter

0 Karma

CommunityUser
Splunk Employee
Splunk Employee

Hi Peter,

We did upgrade the JDK to version "1.8.0_212" but still the JVM is crashing.

Do you have any other inputs on troubleshooting/fixing this problem.

Also one thing to notice is that some times Instance also restarts when we try to take the thread dump using the agent files.

Regards

Vikas Bhatia

0 Karma

Peter_Holditch
Builder

Vikas,

I am beginning to wonder if your scenario is a JVM crash (i.e. the JVM terminated, due to a JVM bug) or a terminal out of memory error due to insifficient headroom in the heap.

For the JVM crash case, you will see a Fatal Error Log.  If you have one of these, please share it.

For the out of memory case, you should see one or more java.lang.OutOfMemory exceptions in the application logs.  Alternatively, if you see no out of memory exceptions logged, you could double confirm that memory headroom is not the root cause by setting the JVM to heap dump on out of memory and making sure there is no heap dump created.

If you do find a fatal error log, you will need to try and match the symptoms in the log with known JDK bugs.  The fix for out of memory would be to increase Xmx to allow additional operating headroom.

Warm regards,

Peter

0 Karma

CommunityUser
Splunk Employee
Splunk Employee

So I tried with the suggested options. here are the results :

  • No Fatal error log file generated, so no jvm crash happened.
  • Tried the -XX:+HeapDumpOnOutOfMemoryError and heap dump was generated when the application crashed and restarted but there are no outofmemory logs in the application logs.

Also i have deployed the application in cloud foundry and cloud foundry also doesn't show the crash reason as out of memory as it should show in case of OOM. But as heap dump is generated, it is giving an indication that OOM is happening but doesn't look like heap.

Suggestions?

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Synthetic Monitoring - Resolved Incident on Detector Alerts

We’ve discovered a bug that affected the auto-clear of Synthetic Detectors in the Splunk Synthetic Monitoring ...

Video | Tom’s Smartness Journey Continues

Remember Splunk Community member Tom Kopchak? If you caught the first episode of our Smartness interview ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud? Learn how unique features like ...