Lowering Stdout Verbosity for AppDynamics Agent and Proxy in Python Applications

[Felipe_Windmoll]

Hi AppDynamics Community,

I'm integrating the AppDynamics Python Agent into a FastAPI project for monitoring purposes and have encountered a bit of a snag regarding log verbosity in my stdout. I'm launching my FastAPI app with the following command to include the AppDynamics agent:

pyagent run -c appdynamics.cfg uvicorn my_app:app --reload

My goal is to reduce the verbosity of the logs from both the AppDynamics agent and the proxy that are output to stdout, aiming to keep my console output clean and focused on more critical issues.

My module versions:

$ pip freeze | grep appdy
appdynamics==23.10.0.6327
appdynamics-bindeps-linux-x64==23.10.0
appdynamics-proxysupport-linux-x64==11.64.3

Here's the content of my `appdynamics.cfg` configuration file:

[agent]
app = my-app
tier = my-tier
node = teste-local-01

[controller]
host = my-controller.saas.appdynamics.com
port = 443
ssl = true
account = my-account
accesskey = my-key

[log]
level = warning
debugging = off

I attempted to decrease the log verbosity further by modifying the `log4j.xml` file for the proxy to set the logging level to WARNING. However, this change didn't have the effect I was hoping for. The `log4j.xml` file I adjusted is located at:

/tmp/appd/lib/cp311-cp311-63ff661bc175896c1717899ca23edc8f5fa87629d9e3bcd02cf4303ea4836f9f/site-packages/appdynamics_bindeps/proxy/conf/logging/log4j.xml

Here are the adjustments I made to the `log4j.xml`:

    <appender class="com.singularity.util.org.apache.log4j.ConsoleAppender" name="ConsoleAppender">
        <layout class="com.singularity.util.org.apache.log4j.PatternLayout">
            <param name="ConversionPattern" value="%d{ABSOLUTE} %5p [%t] %c{1} - %m%n" />
        </layout>
        <filter class="com.singularity.util.org.apache.log4j.varia.LevelRangeFilter">
            <param name="LevelMax" value="FATAL" />
            <param name="LevelMin" value="WARNING" />
        </filter>

Despite these efforts, I'm still seeing a high volume of logs from both the agent and proxy. Could anyone provide guidance or suggestions on how to effectively lower the log output to stdout for both the AppDynamics Python Agent and its proxy? Any tips on ensuring my changes to `log4j.xml` are correctly applied would also be greatly appreciated.

Thank you in advance for your help!

Example of logging messages I would like to remove from my stdout:

2024-03-23 11:15:28,409 [INFO] appdynamics.proxy.watchdog <22759>: Started watchdog with pid=22759
2024-03-23 11:15:28,409 [INFO] appdynamics.proxy.watchdog <22759>: Started watchdog with pid=22759
...
[AD Thread Pool-ProxyControlReq0] Sat Mar 23 11:15:51 BRT 2024[DEBUG]: JavaAgent - Setting AgentClassLoader as Context ClassLoader
[AD Thread Pool-ProxyControlReq0] Sat Mar 23 11:15:52 BRT 2024[INFO]: JavaAgent - Low Entropy Mode: Attempting to swap to non-blocking PRNG algorithm
[AD Thread Pool-ProxyControlReq0] Sat Mar 23 11:15:52 BRT 2024[INFO]: JavaAgent - UUIDPool size is 10
Agent conf directory set to [/home/wsl/.pyenv/versions/3.11.6/lib/python3.11/site-packages/appdynamics_bindeps/proxy/conf]
...
11:15:52,167  INFO [AD Thread Pool-ProxyControlReq0] BusinessTransactions - Starting BT Logs at Sat Mar 23 11:15:52 BRT 2024
11:15:52,168  INFO [AD Thread Pool-ProxyControlReq0] BusinessTransactions - ###########################################################
11:15:52,169  INFO [AD Thread Pool-ProxyControlReq0] BusinessTransactions - Using Proxy Version [Python Agent v23.10.0.6327 (proxy v23.10.0.35234) compatible with 4.5.0.21130 Python Version 3.11.6]
11:15:52,169  INFO [AD Thread Pool-ProxyControlReq0] JavaAgent - Logging set up for log4j2
...
11:15:52,965  INFO [AD Thread Pool-ProxyControlReq0] JDBCConfiguration - Setting normalizePreparedStatements to true
11:15:52,965  INFO [AD Thread Pool-ProxyControlReq0] CallGraphConfigHandler - Call Graph Config Changed  callgraph-granularity-in-ms  Value -null

[iamryan]

Hi @Felipe.Windmoller,

You can submit feature requests on the Idea Exchange or reach out to your AppDynamics CSM. 

[Felipe_Windmoll]

Hello @Ryan.Paredez 

I created the suggestion here - "Create environment variables to pyagent Proxy and Watchdog Log Level" 

Thanks,

Felipe

[iamryan]

Hi @Felipe.Windmoller,

Thanks so much for following up with all this additional information! 

[Felipe_Windmoll]

@Ryan.Paredez 

I would like to make the suggestion of versioning the appdynamics Python module to allow us to pass the Proxy and Watch dog log levels as environment variables.

Thanks

[iamryan]

Hi @Felipe.Windmoller,

It seems the community was not able to jump in and help. Did you happen to find a solution or workaround you can share? 

[Felipe_Windmoll · Accepted solution]

I came across a workaround that was suggested in an unofficial capacity by someone at AppDynamics during their local lab explorations. While this solution isn't officially supported by AppDynamics, it has proven to be effective for adjusting the log levels for both the Proxy and the Watchdog components within my AppDynamics setup. I'd like to share the steps involved, but please proceed with caution and understand that this is not a sanctioned solution.

I recommend changing only the log4j2.xml file, because the proxy messages look like are responsible for almost 99% of the log messages.

Here's a summary of the steps:

• Proxy Log Level: The log4j2.xml file controls this. You can find it within the appdynamics_bindeps module. For example, in my WSL setup, it's located at /home/wsl/.pyenv/versions/3.11.6/lib/python3.11/site-packages/appdynamics_bindeps/proxy/conf/logging/log4j2.xml. In the Docker image python:3.9, the path is /usr/local/lib/python3.9/site-packages/appdynamics_bindeps/proxy/conf/logging/log4j2.xml. Modify the seven log level itens <AsyncLogger> within the <Loggers> section to one of the following: debug, info, warn, error, or fatal.

• Watch Dog Log Level: This can be adjusted in the proxy.py file found within the appdynamics Python module. For example, in my WSL setup, it's located at /home/wsl/.pyenv/versions/3.11.6/lib/python3.11/site-packages/appdynamics/scripts/pyagent/commands/proxy.py. In the Docker image python:3.9, the path is /usr/local/lib/python3.9/site-packages/appdynamics/scripts/pyagent/commands/proxy.py. You will need to hardcode the log level in the configure_proxy_logger and configure_watchdog_logger functions by changing the level variable.

My versions

$ pip freeze | grep appdynamics
appdynamics==24.2.0.6567
appdynamics-bindeps-linux-x64==24.2.0
appdynamics-proxysupport-linux-x64==11.68.3

log4j2.xml

    <Loggers>
        <AsyncLogger name="com.singularity" level="info" additivity="false">
            <AppenderRef ref="Default"/>
            <AppenderRef ref="RESTAppender"/>
            <AppenderRef ref="Console"/>
        </AsyncLogger>
        <AsyncLogger name="com.singularity.dynamicservice" level="info" additivity="false">
            <AppenderRef ref="DynamicServiceAppender"/>
        </AsyncLogger>
        <AsyncLogger name="com.singularity.ee.service.datapipeline" level="info" additivity="false">
            <AppenderRef ref="DataPipelineAppender"/>
            <AppenderRef ref="Console"/>
        </AsyncLogger>
        <AsyncLogger name="com.singularity.datapipeline" level="info" additivity="false">
            <AppenderRef ref="DataPipelineAppender"/>
            <AppenderRef ref="Console"/>
        </AsyncLogger>
        <AsyncLogger name="com.singularity.BCTLogger" level="info" additivity="false" >
            <AppenderRef ref="BCTAppender"/>
        </AsyncLogger>
        <AsyncLogger name ="com.singularity.api" level="info" additivity="false">
            <AppenderRef ref="APIAppender"/>
        </AsyncLogger>
        <AsyncLogger name="com.singularity.segment.TxnTracer" level="info" additivity="false">
            <AppenderRef ref="Default"/>
            <AppenderRef ref="Console"/>
        </AsyncLogger>
        <Root level="error">
            <AppenderRef ref="Console"/>
            <AppenderRef ref="Default"/>
        </Root>
    </Loggers>

proxy.py

def configure_proxy_logger(debug):
    logger = logging.getLogger('appdynamics.proxy')
    level = logging.DEBUG if debug else logging.INFO
    pass

def configure_watchdog_logger(debug):
    logger = logging.getLogger('appdynamics.proxy')
    level = logging.DEBUG if debug else logging.INFO
    pass

Suggestion

Change the hardcoded variables by environment variables:

log4j2.xml file, change from [level="info"] to

proxy.py file, change from [logging.DEBUG if debug else logging.INFO] to

[os.getenv("APP_APPD_WATCHDOG_LOG_LEVEL", "info").upper()]

Warning
Please note, these paths and methods may vary based on your AppDynamics version and environment setup. Always backup files before making changes and be aware that updates to AppDynamics may overwrite your customizations.

I hope this helps!

[Felipe_Windmoll]

Here on StackOverFlow  I share a script to change log4j2.xml and proxy.py automatically.

Besides the script, I put a tip on how to do this inside a Dockerfile.

Basically, I replace the hardcoded values on log4j2.xml and proxy.py by environment variables that allow you to change them more easily on deploy.

log4j2.xml with environment variable

proxy.py with environment variable

def configure_proxy_logger(debug):
    logger = logging.getLogger('appdynamics.proxy')
    level = os.getenv("APP_APPD_WATCHDOG_LOG_LEVEL", "info").upper()
    pass

def configure_watchdog_logger(debug):
    logger = logging.getLogger('appdynamics.proxy')
    level = os.getenv("APP_APPD_WATCHDOG_LOG_LEVEL", "info").upper()
    pass