Splunk AppDynamics

AppDynamics Magine Agent holding deleted files open

CommunityUser
Splunk Employee
Splunk Employee

Good morning. My employer uses appd on all our java app servers, I work in the operations dept, and recently I discovered a rather disturbing trend. Way too many of our servers are getting tickets for disk utilization being above 84%, and it turns out the root cause is that the appd_agent service is holding open dozens or hundreds of huge deleted application log files in /var/log. Bouncing the appd_agent service has now become my first action whenever presented with high disk utilization. It seems to be more common on hosts running Apache Camel, as the busier hosts can roll over logfiles in a matter of seconds.

I don't know what appd_agent is doing with those logfiles. I cannot imagine a reason that a performance monitoring service would need to ever open an application logfile. Looking for errors, I'm sure. It may merely be a local configuration issue, causing those files to be scanned, but I'm not in the loop as to how appd is configured.

Labels (1)
0 Karma
1 Solution

Peter_Holditch
Builder

Patrick,

By itself the machine agent will not touch any logs other than its own.  It is almost certain that someone has configured a machine agent extension of some kind which is opening the logs.

I am afraid your only course of action is to track down whoever set up the machine agent and work with them.

Warm regards,
Peter

View solution in original post

Peter_Holditch
Builder

Patrick,

By itself the machine agent will not touch any logs other than its own.  It is almost certain that someone has configured a machine agent extension of some kind which is opening the logs.

I am afraid your only course of action is to track down whoever set up the machine agent and work with them.

Warm regards,
Peter

Get Updates on the Splunk Community!

CX Day is Coming!

Customer Experience (CX) Day is on October 7th!! We're so excited to bring back another day full of wonderful ...

Strengthen Your Future: A Look Back at Splunk 10 Innovations and .conf25 Highlights!

The Big One: Splunk 10 is Here!  The moment many of you have been waiting for has arrived! We are thrilled to ...

Now Offering the AI Assistant Usage Dashboard in Cloud Monitoring Console

Today, we’re excited to announce the release of a brand new AI assistant usage dashboard in Cloud Monitoring ...