Splunk AppDynamics

AppDynamics Magine Agent holding deleted files open

CommunityUser
Splunk Employee
Splunk Employee

Good morning. My employer uses appd on all our java app servers, I work in the operations dept, and recently I discovered a rather disturbing trend. Way too many of our servers are getting tickets for disk utilization being above 84%, and it turns out the root cause is that the appd_agent service is holding open dozens or hundreds of huge deleted application log files in /var/log. Bouncing the appd_agent service has now become my first action whenever presented with high disk utilization. It seems to be more common on hosts running Apache Camel, as the busier hosts can roll over logfiles in a matter of seconds.

I don't know what appd_agent is doing with those logfiles. I cannot imagine a reason that a performance monitoring service would need to ever open an application logfile. Looking for errors, I'm sure. It may merely be a local configuration issue, causing those files to be scanned, but I'm not in the loop as to how appd is configured.

Labels (1)
0 Karma
1 Solution

Peter_Holditch
Builder

Patrick,

By itself the machine agent will not touch any logs other than its own.  It is almost certain that someone has configured a machine agent extension of some kind which is opening the logs.

I am afraid your only course of action is to track down whoever set up the machine agent and work with them.

Warm regards,
Peter

View solution in original post

Peter_Holditch
Builder

Patrick,

By itself the machine agent will not touch any logs other than its own.  It is almost certain that someone has configured a machine agent extension of some kind which is opening the logs.

I am afraid your only course of action is to track down whoever set up the machine agent and work with them.

Warm regards,
Peter

Get Updates on the Splunk Community!

Fall Into Learning with New Splunk Education Courses

Every month, Splunk Education releases new courses to help you branch out, strengthen your data science roots, ...

Super Optimize your Splunk Stats Searches: Unlocking the Power of tstats, TERM, and ...

By Martin Hettervik, Senior Consultant and Team Leader at Accelerate at Iver, Splunk MVPThe stats command is ...

How Splunk Observability Cloud Prevented a Major Payment Crisis in Minutes

Your bank's payment processing system is humming along during a busy afternoon, handling millions in hourly ...