AppD Archive

agents in containers are not deleted and left registered when the container is replaced

CommunityUser
Splunk Employee
Splunk Employee

I'm testing out the free trial to evaluate the APM features. My apps are Java apps running in docker containers on AWS ECS. I have set up an jvm app agent in each container to start up with the app when they are created. Deployments replace the all containers with zero downtime deployments, and i am using a 'chaos monkey' like system to randomly kill containers to test there is no downtime.

I have noticed with app agent that when a new app agent start, the old one is not deleted, which leaves the node count higher than it is, and also uses up licenses. I haven't been able to work out from the documentation if this is the behaviour I should be expecting to happen, or if there is some set up I have not found to make sure the agents are deregistered when the containers are stopped/killed?

Any help on this would be really useful.

0 Karma

Jean-Baptiste_M
Builder
Hello Stuart,
Have a look here
https://docs.appdynamics.com/display/PRO42/Historical+and+Disconnected+Nodes
for the standard rules (valid for "normal" agents as well as "dockerized"
agents. I'm sure you will find it interesting.

Now, if you want to delete a node once you kill the docker container, you
can use the API (REST) to mark this node as historical
https://docs.appdynamics.com/display/PRO42/Configuration+API#ConfigurationAPI-MarkNodesasHistorical
to free up the license.

Does it make sense?
Best
JB

CommunityUser
Splunk Employee
Splunk Employee

Hi J.B. thank you for your response and the useful links.

From the notes on licensing I was interested by this statement:

For licensing purposes, the Controller releases the license for the agent if the Controller has not received data from the agent in the previous 5 minutes. This license availability behavior is not affected by the historical node status or node deletion timeout settings.

This sounds like there is a hard limit on when licenses are released. Would it therefore be the case that if I had a cluster of 3 apps in containers, and wanted to do a blue/green deployments such that I brought up 3 new instances of the app, and then took down the old 3 when the new are stable, that I would need to have 6 licences?

0 Karma

Jean-Baptiste_M
Builder
My pleasure to help :winking_face:

Indeed in a such scenario, you need 3 more licenses = 6 in total, because
for a certain amount of time, you would have 6 agents connected and
monitoring at the same time.

JB
0 Karma
Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...