Splunk AppDynamics

AppDynamics Magine Agent holding deleted files open

CommunityUser
Splunk Employee
Splunk Employee

Good morning. My employer uses appd on all our java app servers, I work in the operations dept, and recently I discovered a rather disturbing trend. Way too many of our servers are getting tickets for disk utilization being above 84%, and it turns out the root cause is that the appd_agent service is holding open dozens or hundreds of huge deleted application log files in /var/log. Bouncing the appd_agent service has now become my first action whenever presented with high disk utilization. It seems to be more common on hosts running Apache Camel, as the busier hosts can roll over logfiles in a matter of seconds.

I don't know what appd_agent is doing with those logfiles. I cannot imagine a reason that a performance monitoring service would need to ever open an application logfile. Looking for errors, I'm sure. It may merely be a local configuration issue, causing those files to be scanned, but I'm not in the loop as to how appd is configured.

Labels (1)
0 Karma
1 Solution

Peter_Holditch
Builder

Patrick,

By itself the machine agent will not touch any logs other than its own.  It is almost certain that someone has configured a machine agent extension of some kind which is opening the logs.

I am afraid your only course of action is to track down whoever set up the machine agent and work with them.

Warm regards,
Peter

View solution in original post

Peter_Holditch
Builder

Patrick,

By itself the machine agent will not touch any logs other than its own.  It is almost certain that someone has configured a machine agent extension of some kind which is opening the logs.

I am afraid your only course of action is to track down whoever set up the machine agent and work with them.

Warm regards,
Peter

Get Updates on the Splunk Community!

Splunk Mobile: Your Brand-New Home Screen

Meet Your New Mobile Hub  Hello Splunk Community!  Staying connected to your data—no matter where you are—is ...

Introducing Value Insights (Beta): Understand the Business Impact your organization ...

Real progress on your strategic priorities starts with knowing the business outcomes your teams are delivering ...

Enterprise Security (ES) Essentials 8.3 is Now GA — Smarter Detections, Faster ...

As of today, Enterprise Security (ES) Essentials 8.3 is now generally available, helping SOC teams simplify ...