Universal Prometheus to Splunk AppDynamics Integration: Monitor ANY Application with Zero Code Changes

Transform Your Prometheus Metrics into AppDynamics Custom Metrics in Minutes

As organizations adopt cloud-native architectures, monitoring becomes increasingly complex. You might have applications exposing Prometheus metrics, but your enterprise standardizes on AppDynamics for observability, alerting, and business dashboards.

What if you could monitor ANY Prometheus-enabled application in AppDynamics without changing a single line of code?

In this article, I’ll show you how to build a Universal Prometheus to AppDynamics converter that automatically transforms any Prometheus metrics into AppDynamics Custom Metrics, complete with organized hierarchies and human-readable names.

The Challenge: Two Monitoring Worlds

The Prometheus World

Cloud-native standard for metrics
Pull-based model with time-series data
Flexible labeling system
Developer-friendly metric naming (snake_case)

The AppDynamics World

Enterprise monitoring platform
Business context and correlation
Advanced alerting and dashboards
Hierarchical metric organization

The gap? No native way to import Prometheus metrics into AppDynamics while preserving the rich label context and maintaining organized metric hierarchies.

The Solution: Universal Converter

Our solution uses AppDynamics Machine Agent Custom Extensions to:

✅ Automatically discover all Prometheus metrics
✅ Convert technical names to business-friendly format
✅ Preserve label context as hierarchical paths
✅ Zero application changes required
✅ Works with ANY Prometheus endpoint

Architecture Overview

┌─────────────────┐    ┌──────────────────┐    ┌────────────────────┐
│   Application   │    │   Machine Agent  │    │   AppDynamics     │
│                 │    │                  │    │   Controller      │
│ Prometheus      │───▶│ Universal        │───▶│                   │
│ /metrics        │    │ Converter        │    │ Custom Metrics    │
│ Endpoint        │    │ Extension        │    │ Dashboards        │
└─────────────────┘    └──────────────────┘    └────────────────────┘

Data Flow:

Application exposes metrics at /actuator/prometheus
Machine Agent extension scrapes metrics every 60 seconds
Universal converter transforms format and creates hierarchy
AppDynamics Controller receives organized custom metrics
Business users create dashboards and alerts

Step 1: Clone Repository and Start Sample Application

Our sample application is a comprehensive e-commerce microservice that demonstrates realistic Prometheus metrics for AppDynamics integration.

Install and Run

# Clone the repository  
git clone https://github.com/Abhimanyu9988/universal-prometheus-appdynamics
cd universal-prometheus-appdynamics

# Install dependencies
pip install -r requirements.txt

# Start the sample application
python sample-app/microservice.py

Application will start with:

Metrics endpoint: http://localhost:8080/actuator/prometheus
Health check: http://localhost:8080/health
Order API: POST http://localhost:8080/api/orders
User sessions: POST http://localhost:8080/api/users/<id>/session
Cart abandonment: POST http://localhost:8080/api/cart/abandon

Complete E-commerce Sample Application

Our sample application simulates a realistic e-commerce microservice with comprehensive business and infrastructure metrics:

Key Features:

Business Metrics: Orders, revenue, user activity, conversion rates
Infrastructure Metrics: Database connections, cache performance, memory/CPU usage
API Endpoints: Order processing, user sessions, cart abandonment tracking
Background Simulation: Realistic metric generation with temporal patterns

Quick Start:

# Clone the repository
git clone https://github.com/Abhimanyu9988/universal-prometheus-appdynamics
cd universal-prometheus-appdynamics

# Install dependencies
pip install -r requirements.txt

# Start the sample application
python sample-app/microservice.py

Sample Metrics Generated:

Curl below

http://localhost:8080/actuator/prometheus

to see metrics like:

# Business Metrics
orders_processed_total{product_category="electronics",region="us_east",payment_method="credit_card"} 15
order_value_dollars{product_category="electronics",customer_tier="gold"} 750
active_users_current{user_type="premium",subscription_tier="pro"} 45
conversion_rate_percentage{traffic_source="organic",product_category="electronics"} 6.2

# Infrastructure Metrics  
database_connections_active{database_name="user_db",connection_type="read"} 12
cache_hits_total{cache_type="redis",cache_key_pattern="user_*"} 1247
memory_usage_bytes{memory_type="heap"} 856000000
api_requests_total{method="POST",endpoint="/api/orders",status_code="200"} 89

Generate Test Data:

# Create orders
curl -X POST http://localhost:8080/api/orders
curl -X POST http://localhost:8080/api/orders

# User sessions  
curl -X POST http://localhost:8080/api/users/user123/session

# Cart abandonment
curl -X POST http://localhost:8080/api/cart/abandon

The complete application code is available in the GitHub repository at sample-app/microservice.py with additional features including:

Comprehensive metric types: Counters, gauges, histograms, and summaries
Realistic business scenarios: E-commerce order processing, user activity, payment methods
Infrastructure monitoring: Database connections, cache performance, memory/CPU usage
Background simulation: Automatic metric generation with realistic patterns
API endpoints: Interactive endpoints to generate test data
Production patterns: Proper error handling, logging, and metric organization

📁 Full Implementation: View on GitHub

Run the Application

python3 sample_microservice.py

Visit http://localhost:8080/actuator/prometheus to see metrics like:

# HELP orders_processed_total Total orders processed
# TYPE orders_processed_total counter
orders_processed_total{product_type="electronics",region="us-east"} 15.0
orders_processed_total{product_type="clothing",region="europe"} 8.0

# HELP active_users_current Current number of active users
# TYPE active_users_current gauge  
active_users_current{user_type="premium"} 45.0
active_users_current{user_type="standard"} 78.0

Step 2: Universal Prometheus to AppDynamics Converter

Now let’s create the Universal Converter that transforms ANY Prometheus metrics into AppDynamics format:

The Magic: Conversion Logic

The Universal Converter transforms any Prometheus metrics into AppDynamics format. Here’s how it works with our sample application:

Input (Prometheus Format):

orders_processed_total{product_category="electronics",region="us_east",payment_method="credit_card"} 15
active_users_current{user_type="premium",subscription_tier="pro"} 45  
database_connections_active{database_name="user_db",connection_type="read"} 12

Output (AppDynamics Format):

name=Custom Metrics|App|Orders Processed Total|Product Category Electronics|Region Us East|Payment Method Credit Card,value=15,aggregator=OBSERVATION,time-rollup=CURRENT,cluster-rollup=COLLECTIVE
name=Custom Metrics|App|Active Users Current|User Type Premium|Subscription Tier Pro,value=45,aggregator=OBSERVATION,time-rollup=CURRENT,cluster-rollup=INDIVIDUAL
name=Custom Metrics|App|Database Connections Active|Database Name User Db|Connection Type Read,value=12,aggregator=OBSERVATION,time-rollup=CURRENT,cluster-rollup=INDIVIDUAL

Core Conversion Functions:

Complete code :

#!/bin/bash
# Universal Prometheus to AppDynamics Converter
# Transforms: orders_processed_total{product_type="electronics"} 15
# Into: Custom Metrics|App|Orders Processed Total|Product Type Electronics,value=15
PROMETHEUS_ENDPOINT="http://localhost:8080/actuator/prometheus"
MAX_METRICS=100
# Convert snake_case to Title Case
convert_to_title_case() {
    echo "$1" | sed 's/_/ /g' | sed 's/\b\w/\U&/g'
}
# Clean label values for AppDynamics hierarchy  
clean_label_value() {
    echo "$1" | sed 's/"//g' | sed 's/[^a-zA-Z0-9.]/ /g' | sed 's/\b\w/\U&/g'
}
# Process each metric line
process_metric_line() {
    local line="$1"
    
    # Parse: metric_name{labels} value
    if [[ "$line" =~ ^([a-zA-Z_][a-zA-Z0-9_]*)(.*)[[:space:]]+([0-9.]+)$ ]]; then
        local metric_name="${BASH_REMATCH[1]}"
        local labels_part="${BASH_REMATCH[2]}" 
        local value="${BASH_REMATCH[3]}"
        
        # Convert to human readable
        local readable_name=$(convert_to_title_case "$metric_name")
        local metric_path="Custom Metrics|App|$readable_name"
        
        # Process labels into hierarchy
        if [[ "$labels_part" =~ ^\{.*\}$ ]]; then
            labels_part="${labels_part#\{}"
            labels_part="${labels_part%\}}"
            
            IFS=',' read -ra LABELS <<< "$labels_part"
            for label in "${LABELS[@]}"; do
                if [[ "$label" =~ ^[[:space:]]*([^=]+)=[[:space:]]*\"?([^\"]+)\"?$ ]]; then
                    local key=$(convert_to_title_case "${BASH_REMATCH[1]}")
                    local val=$(clean_label_value "${BASH_REMATCH[2]}")
                    metric_path="$metric_path|$key $val"
                fi
            done
        fi
        
        # Output AppDynamics format
        echo "name=$metric_path,value=${value%.*},aggregator=OBSERVATION,time-rollup=CURRENT,cluster-rollup=INDIVIDUAL"
    fi
}
# Main execution
METRICS_DATA=$(curl -s --max-time 30 "$PROMETHEUS_ENDPOINT")
if [ $? -ne 0 ]; then
    echo "name=Custom Metrics|App|Connection Status,value=0,aggregator=OBSERVATION,time-rollup=CURRENT,cluster-rollup=INDIVIDUAL"
    exit 1
fi
echo "name=Custom Metrics|App|Connection Status,value=1,aggregator=OBSERVATION,time-rollup=CURRENT,cluster-rollup=INDIVIDUAL"
# Process all metrics
metric_count=0
while IFS= read -r line && [ $metric_count -lt $MAX_METRICS ]; do
    result=$(process_metric_line "$line")
    if [ ! -z "$result" ]; then
        echo "$result"
        ((metric_count++))
    fi
done <<< "$METRICS_DATA"

What This Creates in AppDynamics

Before (Prometheus):

orders_processed_total{product_category="electronics",region="us_east",payment_method="credit_card"} 15
active_users_current{user_type="premium",subscription_tier="pro"} 45
database_connections_active{database_name="user_db",connection_type="read"} 12

After (AppDynamics Hierarchy):

Custom Metrics
└── App
    ├── Orders Processed Total
    │   └── Product Category Electronics
    │       └── Region Us East
    │           └── Payment Method Credit Card = 15
    ├── Active Users Current
    │   └── User Type Premium
    │       └── Subscription Tier Pro = 45
    └── Database Connections Active
        └── Database Name User Db
            └── Connection Type Read = 12

Step 3: Deploy to AppDynamics Machine Agent

Create Extension Directory

mkdir -p /opt/appdynamics/machine-agent/monitors/UniversalPrometheusMonitor
cd /opt/appdynamics/machine-agent/monitors/UniversalPrometheusMonitor

Save the Converter Script

# Save the full converter script
cat > UniversalPrometheusMonitor.sh << 'EOF'
[Full script content from above]
EOF

chmod +x UniversalPrometheusMonitor.sh

Create monitor.xml

<?xml version="1.0" encoding="UTF-8"?>
<monitor>
    <name>UniversalPrometheusMonitor</name>
    <type>managed</type>
    <description>Universal Prometheus to AppDynamics Converter</description>
    <monitor-run-task>
        <execution-style>periodic</execution-style>
        <execution-frequency-in-seconds>60</execution-frequency-in-seconds>
        <name>Universal Prometheus Collection</name>
        <type>executable</type>
        <execution-timeout-in-secs>60</execution-timeout-in-secs>
        <task-arguments>
        </task-arguments>
        <executable-task>
            <type>file</type>
            <file>UniversalPrometheusMonitor.sh</file>
        </executable-task>
    </monitor-run-task>
</monitor>

Test and Deploy

# Test manually
./UniversalPrometheusMonitor.sh

# Restart Machine Agent
sudo systemctl restart machine-agent

Step 4: Verify in AppDynamics Controller

After 2–3 minutes, navigate to:

Metric Browser → Application Infrastructure Performance → [Machine Agent] → Custom Metrics → App

You’ll see organized metrics like:

Orders Processed Total
Product Type Electronics → Region Us East
Product Type Clothing → Region Europe
Active Users Current
User Type Premium
User Type Standard
Database Connections Active
Database Name User Db
Database Name Order Db

Real-World Applications

This solution works perfectly for:

🏗️ Cloud-Native Applications

Kubernetes deployments with Prometheus metrics
Microservices exposing custom business metrics
Container workloads with infrastructure metrics

🔧 Enterprise Integration

Camunda workflow engines (both Platform 7 & 😎
Apache Kafka clusters
Elasticsearch deployments
Custom applications with Prometheus client libraries

📊 Business Intelligence

E-commerce metrics (orders, revenue, cart abandonment)
SLA monitoring (response times, error rates)
Capacity planning (resource utilization, growth trends)

Advanced Configuration

Customize Metric Filtering

# Filter specific metrics
INCLUDE_PATTERNS="orders_|users_|database_"
EXCLUDE_PATTERNS="_bucket|_sum|_count"
if [[ "$metric_name" =~ $INCLUDE_PATTERNS ]] && [[ ! "$metric_name" =~ $EXCLUDE_PATTERNS ]]; then
    # Process metric
fi

Add Business Context

# Map technical names to business terms
case "$metric_name" in
    "orders_processed_total") readable_name="Revenue Orders";;
    "active_users_current") readable_name="Online Customers";;
    "database_connections_active") readable_name="DB Performance";;
    *) readable_name=$(convert_to_title_case "$metric_name");;
esac

Configure Aggregation Types

# Set appropriate aggregation based on metric type
if [[ "$metric_name" =~ _total$ ]]; then
    aggregator="OBSERVATION"
    cluster_rollup="COLLECTIVE"
elif [[ "$metric_name" =~ _current$ ]]; then
    aggregator="OBSERVATION" 
    cluster_rollup="INDIVIDUAL"
fi

Monitoring and Troubleshooting

Health Checks

The converter includes built-in monitoring:

# Connection status
name=Custom Metrics|App|Connection Status,value=1

# Metrics processed
name=Custom Metrics|App|Monitoring|Total Metrics Processed,value=67

Debugging

# Test the converter manually
./UniversalPrometheusMonitor.sh | head -20

# Check Machine Agent logs
tail -f /opt/appdynamics/machine-agent/logs/machine-agent.log | grep Universal

# Verify endpoint connectivity
curl http://localhost:8080/actuator/prometheus | head -10

Performance Considerations

Metric Volume Management

MAX_METRICS=100  # Prevent overwhelming AppDynamics
BATCH_SIZE=50    # Process in batches for large metric sets

Memory Usage

# For high-volume environments
TIMEOUT=10       # Shorter timeouts
FREQUENCY=120    # Less frequent collection (2 minutes)

Network Optimization

# Use local endpoints when possible
PROMETHEUS_ENDPOINT="http://localhost:8080/actuator/prometheus"

# Add retry logic
for i in {1..3}; do
    METRICS_DATA=$(curl -s --max-time $TIMEOUT "$PROMETHEUS_ENDPOINT")
    [ $? -eq 0 ] && break
    sleep 5
done

Security Best Practices

Authentication

# For secured Prometheus endpoints
curl -H "Authorization: Bearer $TOKEN" "$PROMETHEUS_ENDPOINT"

# Or with basic auth
curl -u "$USERNAME:$PASSWORD" "$PROMETHEUS_ENDPOINT"

SSL/TLS

# For HTTPS endpoints
curl --cacert /path/to/ca.crt "$PROMETHEUS_ENDPOINT"

Network Security

✅ Restrict access to Prometheus endpoints
✅ Use internal networks for metric collection
✅ Monitor extension logs for security events
✅ Rotate credentials regularly

Scaling Across Environments

Multi-Environment Setup

# Development
PROMETHEUS_ENDPOINT="http://dev-app:8080/actuator/prometheus"
METRIC_PREFIX="Custom Metrics|Dev|App"

# Production  
PROMETHEUS_ENDPOINT="http://prod-app:8080/actuator/prometheus"
METRIC_PREFIX="Custom Metrics|Prod|App"

Multiple Applications

Create separate extensions for each application:

monitors/
├── App1PrometheusMonitor/
├── App2PrometheusMonitor/
└── CamundaPrometheusMonitor/

Conclusion

This Universal Prometheus to AppDynamics converter bridges the gap between cloud-native monitoring and enterprise observability platforms.

Key Benefits:

✅ Zero code changes to existing applications
✅ Automatic metric discovery and organization
✅ Human-readable hierarchies in AppDynamics
✅ Enterprise-grade monitoring with business context
✅ Works with ANY Prometheus endpoint

Perfect For:

DevOps teams adopting cloud-native technologies
Enterprise architects standardizing on AppDynamics
SRE engineers needing comprehensive observability
Business stakeholders requiring custom dashboards

The solution is production-ready, scalable, and maintainable — giving you the best of both worlds: Prometheus’s flexibility with AppDynamics’s enterprise capabilities.

Resources

📁 GitHub Repository: universal-prometheus-appdynamics
📖 AppDynamics Documentation: Machine Agent Extensions
🔧 Prometheus Client Libraries: prometheus.io/docs/instrumenting/clientlibs

Found this helpful? ⭐ Star the repository and share your success stories in the comments below!

Have questions? 💬 Join the discussion and let’s build better monitoring solutions together.

This article is part of a series on enterprise observability and monitoring best practices. Follow for more practical guides on AppDynamics, Prometheus, and modern monitoring architectures.

Are you a member of the Splunk Community?

Universal Prometheus to Splunk AppDynamics Integration: Monitor ANY Application with Zero Code Changes