Splunk AppDynamics

AppDynamics Magine Agent holding deleted files open

CommunityUser
Splunk Employee
Splunk Employee

Good morning. My employer uses appd on all our java app servers, I work in the operations dept, and recently I discovered a rather disturbing trend. Way too many of our servers are getting tickets for disk utilization being above 84%, and it turns out the root cause is that the appd_agent service is holding open dozens or hundreds of huge deleted application log files in /var/log. Bouncing the appd_agent service has now become my first action whenever presented with high disk utilization. It seems to be more common on hosts running Apache Camel, as the busier hosts can roll over logfiles in a matter of seconds.

I don't know what appd_agent is doing with those logfiles. I cannot imagine a reason that a performance monitoring service would need to ever open an application logfile. Looking for errors, I'm sure. It may merely be a local configuration issue, causing those files to be scanned, but I'm not in the loop as to how appd is configured.

Labels (1)
0 Karma
1 Solution

Peter_Holditch
Builder

Patrick,

By itself the machine agent will not touch any logs other than its own.  It is almost certain that someone has configured a machine agent extension of some kind which is opening the logs.

I am afraid your only course of action is to track down whoever set up the machine agent and work with them.

Warm regards,
Peter

View solution in original post

Peter_Holditch
Builder

Patrick,

By itself the machine agent will not touch any logs other than its own.  It is almost certain that someone has configured a machine agent extension of some kind which is opening the logs.

I am afraid your only course of action is to track down whoever set up the machine agent and work with them.

Warm regards,
Peter

Get Updates on the Splunk Community!

Splunk Observability Synthetic Monitoring - Resolved Incident on Detector Alerts

We’ve discovered a bug that affected the auto-clear of Synthetic Detectors in the Splunk Synthetic Monitoring ...

Video | Tom’s Smartness Journey Continues

Remember Splunk Community member Tom Kopchak? If you caught the first episode of our Smartness interview ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud? Learn how unique features like ...