Splunk AppDynamics

BTs and snapshots are not being captured

Sandeep_Poonia
New Member

I have a spring boot application, deployed in AKS. I am using Appynamics docker image to inject the agent into JVM. The setup is working fine in lower environments. Even in the production environment, everything worked fine for 1-2 days. Then I configured the transaction detection rules of java to include a custom header in the transaction name, making them similar to what we have in non-production environments. It worked for a few hours, then it stopped capturing any business transactions and transaction snapshots. I can still see the traffic using the Service Endpoints, where the number of calls and the average response time is captured.

I tried to look at the events, and there I can see some events like

  • New Business Transaction Discovered /xxx.yyy (But no data for this in Business Transaction Tab)
  • Errors like - Application Server Exception
  • Agent Data Diagnostics - OVERFLOW_BT

Currently, the production environment has less traffic than the non-production environment. And apart from this application, we only have a few Browser and Mobile apps, but no other Java/.Net application.

The application has not been restarted since I configured the business transaction name. Tried setting "find-entry-points"  to true, but could not make much sense from its logs.

Labels (1)
0 Karma

Morelz
Builder

Hi Sandeep

I am assuming you have not reached your Business transaction limit already and hence no new BT's are being detected, the limit by default 200? Otherwise look at the below

If the transaction detection rule initially worked and then stopped, initiate a debug logging session for the node for 5 minutes and have a look at the logs.

In most cases this is due to some type of limit that has been reached which then in some instances completely stops the monitoring of BT's until the agent is reset or the JVM is restarted.

You can look in the logs for errors like this, where you find the keyword "exceeded"

WARN BoundsEnforcer - BoundsEnforcer for collection com.singularity.ee.agent.util.bounded.collections.BoundedConcurrentReferenceHashMap with service name TransactionMonitoringService and property name SqlCacheHolder-connectionVsZosSSID-limit has exceeded maximum size of 3300 entries

Have a look at the logs and you will most likely find which limit is being reached and then depending on what limit it is, either increase the limit or resolve the problem on why it's causing the limit to be reached.

Ciao

Get Updates on the Splunk Community!

Fueling your curiosity with new Splunk ILT and eLearning courses

At Splunk Education, we’re driven by curiosity—both ours and yours! That’s why we’re committed to delivering ...

Splunk AI Assistant for SPL 1.1.0 | Now Personalized to Your Environment for Greater ...

Splunk AI Assistant for SPL has transformed how users interact with Splunk, making it easier than ever to ...

Unleash Unified Security and Observability with Splunk Cloud Platform

     Now Available on Microsoft AzureOn Demand Now Step boldly into the AI revolution with enhanced security ...