Splunk AppDynamics

AppDynamics Magine Agent holding deleted files open

CommunityUser
Splunk Employee
Splunk Employee

Good morning. My employer uses appd on all our java app servers, I work in the operations dept, and recently I discovered a rather disturbing trend. Way too many of our servers are getting tickets for disk utilization being above 84%, and it turns out the root cause is that the appd_agent service is holding open dozens or hundreds of huge deleted application log files in /var/log. Bouncing the appd_agent service has now become my first action whenever presented with high disk utilization. It seems to be more common on hosts running Apache Camel, as the busier hosts can roll over logfiles in a matter of seconds.

I don't know what appd_agent is doing with those logfiles. I cannot imagine a reason that a performance monitoring service would need to ever open an application logfile. Looking for errors, I'm sure. It may merely be a local configuration issue, causing those files to be scanned, but I'm not in the loop as to how appd is configured.

Labels (1)
0 Karma
1 Solution

Peter_Holditch
Builder

Patrick,

By itself the machine agent will not touch any logs other than its own.  It is almost certain that someone has configured a machine agent extension of some kind which is opening the logs.

I am afraid your only course of action is to track down whoever set up the machine agent and work with them.

Warm regards,
Peter

View solution in original post

Peter_Holditch
Builder

Patrick,

By itself the machine agent will not touch any logs other than its own.  It is almost certain that someone has configured a machine agent extension of some kind which is opening the logs.

I am afraid your only course of action is to track down whoever set up the machine agent and work with them.

Warm regards,
Peter

Get Updates on the Splunk Community!

Observe and Secure All Apps with Splunk

  Join Us for Our Next Tech Talk: Observe and Secure All Apps with SplunkAs organizations continue to innovate ...

Splunk Decoded: Business Transactions vs Business IQ

It’s the morning of Black Friday, and your e-commerce site is handling 10x normal traffic. Orders are flowing, ...

Fastest way to demo Observability

I’ve been having a lot of fun learning about Kubernetes and Observability. I set myself an interesting ...