Splunk AppDynamics

BTs and snapshots are not being captured

Sandeep_Poonia
New Member

I have a spring boot application, deployed in AKS. I am using Appynamics docker image to inject the agent into JVM. The setup is working fine in lower environments. Even in the production environment, everything worked fine for 1-2 days. Then I configured the transaction detection rules of java to include a custom header in the transaction name, making them similar to what we have in non-production environments. It worked for a few hours, then it stopped capturing any business transactions and transaction snapshots. I can still see the traffic using the Service Endpoints, where the number of calls and the average response time is captured.

I tried to look at the events, and there I can see some events like

  • New Business Transaction Discovered /xxx.yyy (But no data for this in Business Transaction Tab)
  • Errors like - Application Server Exception
  • Agent Data Diagnostics - OVERFLOW_BT

Currently, the production environment has less traffic than the non-production environment. And apart from this application, we only have a few Browser and Mobile apps, but no other Java/.Net application.

The application has not been restarted since I configured the business transaction name. Tried setting "find-entry-points"  to true, but could not make much sense from its logs.

Labels (1)
0 Karma

Morelz
Builder

Hi Sandeep

I am assuming you have not reached your Business transaction limit already and hence no new BT's are being detected, the limit by default 200? Otherwise look at the below

If the transaction detection rule initially worked and then stopped, initiate a debug logging session for the node for 5 minutes and have a look at the logs.

In most cases this is due to some type of limit that has been reached which then in some instances completely stops the monitoring of BT's until the agent is reset or the JVM is restarted.

You can look in the logs for errors like this, where you find the keyword "exceeded"

WARN BoundsEnforcer - BoundsEnforcer for collection com.singularity.ee.agent.util.bounded.collections.BoundedConcurrentReferenceHashMap with service name TransactionMonitoringService and property name SqlCacheHolder-connectionVsZosSSID-limit has exceeded maximum size of 3300 entries

Have a look at the logs and you will most likely find which limit is being reached and then depending on what limit it is, either increase the limit or resolve the problem on why it's causing the limit to be reached.

Ciao

Get Updates on the Splunk Community!

Splunk AI Assistant for SPL | Key Use Cases to Unlock the Power of SPL

Splunk AI Assistant for SPL | Key Use Cases to Unlock the Power of SPL  The Splunk AI Assistant for SPL ...

Buttercup Games: Further Dashboarding Techniques (Part 5)

This series of blogs assumes you have already completed the Splunk Enterprise Search Tutorial as it uses the ...

Customers Increasingly Choose Splunk for Observability

For the second year in a row, Splunk was recognized as a Leader in the 2024 Gartner® Magic Quadrant™ for ...