We have universal forwarders set up on some machines which are part of an auto-scale group in EC2. In some cases the group scales up during peak load. These auto-scale events are somewhat rare, but often times leave us with more instances running than we need once things simmer down.
To combat this, we are devising a way to terminate instances during our low load times. We can terminate the application that is writing the logs and tell Amazon to terminate the instance. But we need to be sure that the forwarder is "caught up", meaning it has reached the end of each of its monitored files.
Is there a way to tell (other than tcpdump or similar) whether the forwarder is still doing anything? This would need to be a scriptable solution so no human intervention is required.
... View more