Complete Guide: Monitor Google Cloud Pub/Sub with Splunk AppDynamics Machine Agent

Building a production-ready monitoring solution for GCP Pub/Sub that provides comprehensive visibility into your messaging infrastructure

The Challenge

Google Cloud Pub/Sub is the backbone of many modern distributed systems, handling millions of messages daily. But here’s the problem: how do you monitor something you can’t see?

Most organizations struggle with:

❌ Limited visibility into topic and subscription health
❌ Reactive monitoring — finding out about issues after they impact users
❌ Fragmented metrics across different tools and dashboards
❌ Complex setup requiring deep GCP and monitoring expertise

What if I told you there’s a way to get comprehensive Pub/Sub monitoring integrated directly into your existing AppDynamics dashboard with minimal setup?

The Solution: End-to-End Automation

In this guide, we’ll build a complete monitoring solution that:

✅ Automatically creates GCP Pub/Sub topics and subscriptions
✅ Generates realistic test data for immediate monitoring
✅ Collects 50+ metrics covering every aspect of your Pub/Sub infrastructure
✅ Integrates seamlessly with AppDynamics Machine Agent
✅ Supports multiple platforms (AWS, GCP, Azure, on-premises)
✅ Includes cleanup scripts for easy environment management

Architecture Overview

Here’s what we’re building:

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   GCP Pub/Sub   │    │  Metrics Script  │    │   AppDynamics   │
│                 │    │                  │    │                 │
│ ┌─────────────┐ │    │ ┌──────────────┐ │    │ ┌─────────────┐ │
│ │   Topics    │◄┼────┼─│    gcloud    │ │    │ │   Machine   │ │
│ └─────────────┘ │    │ │     CLI      │ │    │ │    Agent    │ │
│                 │    │ └──────────────┘ │    │ └─────────────┘ │
│ ┌─────────────┐ │    │ ┌──────────────┐ │    │        │        │
│ │Subscriptions│◄┼────┼─│ Metrics      │─┼────┼────────┘        │
│ └─────────────┘ │    │ │ Collector    │ │    │                 │
│                 │    │ └──────────────┘ │    │ ┌─────────────┐ │
│ ┌─────────────┐ │    │ ┌──────────────┐ │    │ │ Controller  │ │
│ │   Metrics   │◄┼────┼─│    JSON      │ │    │ │ Dashboard   │ │
│ └─────────────┘ │    │ │    Auth      │ │    │ └─────────────┘ │
└─────────────────┘    │ └──────────────┘ │    └─────────────────┘
                       └──────────────────┘

Prerequisites

Before we start, ensure you have:

GCP Project with billing enabled
AppDynamics Machine Agent installed and running
Linux system (we support Amazon Linux 2/2023, Ubuntu, RHEL/CentOS, Rocky Linux)
Basic command-line knowledge

Don’t worry if you’re missing some pieces — we’ll guide you through everything!

Part 1: Setting Up the Foundation

1.1 Clone the Repository

First, let’s get our toolkit:

git clone https://github.com/Abhimanyu9988/gcp-pubsub-appdynamics.git
cd gcp-pubsub-appdynamics
chmod +x *.sh

1.2 Install Prerequisites

Our smart installer detects your operating system and installs everything needed:

bash

sudo ./ec2-pre-req.sh

What this does:

🔧 Detects your OS (Amazon Linux 2/2023, Ubuntu, RHEL, etc.)
📥 Installs Google Cloud SDK with proper Python version
🛠️ Installs required tools (jq, curl, etc.)
⚙️ Sets up proper permissions and paths
✅ Validates everything works

Sample output:

============================================== 
Multi-Distribution GCP Pub/Sub Prerequisites
==============================================

[INFO] Detected OS: Amazon Linux 2
[SUCCESS] System packages updated
[SUCCESS] Python 3.9 installed
[SUCCESS] ✅ Google Cloud SDK installed successfully
[SUCCESS] 🎉 All prerequisites installed successfully!

Part 2: GCP Service Account Setup

2.1 Create Service Account (One-time Setup)

On your local machine where you have gcloud configured:

bash

./create_service_account.sh

What this creates:

🔐 Service account with minimal required permissions
🗝️ JSON key file for authentication
🛡️ IAM roles: pubsub.viewer, monitoring.viewer, serviceusage.serviceUsageViewer

Sample output:

bash

============================================
GCP Service Account Creation for Pub/Sub Monitoring
============================================

[SUCCESS] Using project: my-gcp-project
[SUCCESS] Created service account: pubsub-monitor@my-project.iam.gserviceaccount.com
[SUCCESS] ✅ Assigned: roles/pubsub.viewer
[SUCCESS] ✅ Service account key created: pubsub-monitor-service-account.json

📦 Transfer to AWS Linux 2:
   scp -i your-key.pem pubsub-monitor-service-account.json ec2-user@your-instance-ip:~/

2.2 Transfer Credentials

Copy the service account key to your monitoring server:

bash

# On your local machine
scp -i your-aws-key.pem pubsub-monitor-service-account.json ec2-user@your-server:~/

# On your monitoring server
sudo mkdir -p /opt/appdynamics
sudo mv ~/pubsub-monitor-service-account.json /opt/appdynamics/
sudo chmod 600 /opt/appdynamics/pubsub-monitor-service-account.json

Part 3: Creating Pub/Sub Resources

3.1 Set Up Your Environment

bash

export GCP_PROJECT_ID="your-actual-project-id"

3.2 Create Topics and Subscriptions

bash

./pubsub_create.sh

What this does:

🎯 Creates Pub/Sub topic: appdynamics-monitoring-topic
📥 Creates subscription: appdynamics-monitoring-subscription
📊 Publishes 100 sample messages with realistic data
📈 Generates custom metrics for immediate monitoring
🔄 Optional: Runs continuous simulation workload

Sample output:

bash

==================================
GCP Pub/Sub Creation Script
==================================

[SUCCESS] Created topic: appdynamics-monitoring-topic
[SUCCESS] Created subscription: appdynamics-monitoring-subscription
[INFO] Publishing 100 messages to appdynamics-monitoring-topic
[SUCCESS] Publishing completed:
  Total messages: 100
  Published successfully: 100
  Failed to publish: 0
  Messages per second: 23

3.3 Verify Resources

bash

./pubsub_info.sh status

This shows you exactly what was created and provides troubleshooting information.

Part 4: Configure Metrics Collection

4.1 Edit the Main Script

Update the configuration in script.sh:

bash

vi script.sh

# Update these values:
PROJECT_ID="your-actual-project-id"
SERVICE_ACCOUNT_KEY_FILE="/opt/appdynamics/pubsub-monitor-service-account.json"

# Optional: Add more topics/subscriptions
TOPIC_NAMES="topic1,topic2,topic3"
SUBSCRIPTION_NAMES="sub1,sub2,sub3"

4.2 Test Metrics Collection

bash

./script.sh

Expected output (50+ metrics):

bash

name=Custom Metrics|PubSub|Health|Collection Success, value=1
name=Custom Metrics|PubSub|Topic|appdynamics-monitoring-topic|Status, value=1
name=Custom Metrics|PubSub|Subscription|appdynamics-monitoring-subscription|Ack Deadline, value=60
name=Custom Metrics|PubSub|API|PubSub Enabled, value=1
name=Custom Metrics|PubSub|Project|Total Topics, value=5

Part 5: AppDynamics Integration

5.1 Install Machine Agent Extension

bash

# Create extension directory
sudo mkdir -p /opt/appdynamics/machine-agent/monitors/PubSubMonitor

# Copy files
sudo cp script.sh /opt/appdynamics/machine-agent/monitors/PubSubMonitor/
sudo cp monitor.xml /opt/appdynamics/machine-agent/monitors/PubSubMonitor/

# Set permissions
sudo chown -R appdynamics:appdynamics /opt/appdynamics/machine-agent/monitors/PubSubMonitor
sudo chmod +x /opt/appdynamics/machine-agent/monitors/PubSubMonitor/script.sh

5.2 Restart Machine Agent

bash

sudo systemctl restart appdynamics-machine-agent

# Verify it's running
sudo systemctl status appdynamics-machine-agent

5.3 Check Logs

bash

# Check Machine Agent logs
tail -f /opt/appdynamics/machine-agent/logs/machine-agent.log

# Check for our extension
grep -i pubsub /opt/appdynamics/machine-agent/logs/machine-agent.log

Part 6: The Metrics Deep Dive

Our solution collects 50+ comprehensive metrics across these categories:

🎯 Topic Metrics

Status & Health: Topic accessibility, configuration validation
Subscription Management: Count and health of attached subscriptions
Security: IAM policy accessibility
Configuration: Message retention settings

📥 Subscription Metrics

Operational Health: Status, acknowledgment deadlines
Configuration Analysis: Push vs Pull detection, retry policies
Advanced Features: Dead letter queues, message filtering
Performance: Message retention, delivery settings

🔌 API & Service Health

Service Availability: Pub/Sub and Monitoring API status
Operational Capabilities: List topics/subscriptions permissions
Health Scoring: Overall API accessibility

📊 Project-Level Insights

Resource Inventory: Total topics and subscriptions across project
Configuration Validation: Setup verification
Custom Metrics Integration: Deployment script metrics

Part 7: Viewing in AppDynamics

7.1 Navigate to Metrics

Log into your AppDynamics Controller
Go to Servers → Machine Agents → Your Server
Navigate to Your Server → Custom Metrics
Look for Custom Metrics | PubSub

Screenshot 2025-08-01 at 12.21.44 AM.png

7.2 Create Dashboards

Create custom dashboards with widgets for:

Health Overview:

Collection Success Rate
GCP Connectivity Status
API Health Score

Topic Performance:

Topic Status by Name
Subscription Count Distribution
Success Rate Trends

Subscription Analytics:

Ack Deadline Distribution
Push vs Pull Breakdown
Dead Letter Queue Usage

Project Insights:

Total Resource Counts
Configuration Compliance
Custom Metrics Trends

Part 8: Production Deployment

8.1 Security Best Practices

bash

# Secure service account keys
sudo chmod 600 /opt/appdynamics/*.json
sudo chown appdynamics:appdynamics /opt/appdynamics/*.json

# Enable audit logging
gcloud logging sinks create pubsub-monitoring-sink \
    bigquery.googleapis.com/projects/YOUR_PROJECT/datasets/audit_logs \
    --log-filter='protoPayload.serviceName="pubsub.googleapis.com"'

8.2 Monitoring Multiple Environments

bash

# Development environment
export GCP_PROJECT_ID="dev-project"
export TOPIC_NAMES="dev-orders,dev-inventory"

# Production environment  
export GCP_PROJECT_ID="prod-project"
export TOPIC_NAMES="orders,inventory,notifications,analytics"

8.3 Alerting Setup

Create AppDynamics policies for:

Topic/Subscription availability < 100%
Collection errors > 0
API health score < 1
Custom metric age > 5 minutes

Part 9: Maintenance & Operations

9.1 Resource Management

bash

# Check current resources
./pubsub_info.sh status

# View available metrics  
./pubsub_info.sh metrics

# Generate more test data
RUN_SIMULATION=true ./pubsub_create.sh

# Clean up everything
./pubsub_destroy.sh

9.2 Service Account Rotation

bash

# Create new service account
./create_service_account.sh

# Test with new credentials
./script.sh

# Delete old service account
./delete_service_account.sh

9.3 Troubleshooting

Common Issues & Solutions:

Authentication Errors:

bash

# Verify service account file
jq empty /opt/appdynamics/pubsub-monitor-service-account.json

# Test authentication
gcloud auth activate-service-account --key-file=/opt/appdynamics/pubsub-monitor-service-account.json

Permission Denied:

bash

# Check IAM roles
gcloud projects get-iam-policy YOUR_PROJECT_ID \
    --flatten="bindings[].members" \
    --filter="bindings.members:your-service-account@project.iam.gserviceaccount.com"

No Metrics in AppDynamics:

bash

# Test script manually
cd /opt/appdynamics/machine-agent/monitors/PubSubMonitor
./script.sh

# Check Machine Agent logs
tail -f /opt/appdynamics/machine-agent/logs/machine-agent.log

Advanced Use Cases

Multi-Project Monitoring

Monitor Pub/Sub across multiple GCP projects by deploying separate instances with different service accounts:

bash

# Project 1 - Development
PROJECT_ID="dev-project" SERVICE_ACCOUNT_KEY_FILE="/opt/appdynamics/dev-sa.json" ./script.sh

# Project 2 - Production  
PROJECT_ID="prod-project" SERVICE_ACCOUNT_KEY_FILE="/opt/appdynamics/prod-sa.json" ./script.sh

Custom Metrics Integration

Extend the solution to include your application-specific metrics:

bash

# Add custom metrics to /tmp/pubsub_custom_metrics.log
echo "[$(date -u +"%Y-%m-%dT%H:%M:%S.%3NZ")] CUSTOM_METRIC order_processing_rate 150" >> /tmp/pubsub_custom_metrics.log

CI/CD Integration

Include in your deployment pipelines:

yaml

# GitHub Actions example
- name: Setup Pub/Sub Monitoring
  run: |
    ./pubsub_create.sh
    ./script.sh
    # Verify metrics collection
    if grep -q "Collection Success, value=1" <(./script.sh); then
      echo "✅ Monitoring setup successful"
    else
      echo "❌ Monitoring setup failed"
      exit 1
    fi

Performance & Scaling

Our solution is designed for production scale:

⚡ Fast Execution: Collects 50+ metrics in under 10 seconds
🔄 Concurrent Safe: Multiple instances can run simultaneously
📊 Scalable: Handles hundreds of topics/subscriptions
💾 Lightweight: Minimal memory and CPU footprint
🛡️ Resilient: Continues partial collection even if some resources fail

Performance Metrics:

Average collection time: 5-7 seconds
Memory usage: < 50MB during collection
Network requests: Optimized with bulk operations
Error recovery: Continues on partial failures

What's Next?

You now have a production-ready Pub/Sub monitoring solution that provides:

✅ Complete Visibility - 50+ metrics covering every aspect
✅ Automated Setup - One-click deployment and configuration
✅ Integration Ready - Native AppDynamics dashboard integration
✅ Production Tested - Handles real-world scale and complexity
✅ Maintenance Friendly - Easy updates, rotation, and cleanup

Take It Further

📊 Create Custom Dashboards with business-specific KPIs
🚨 Set Up Intelligent Alerting for proactive issue detection
🔄 Integrate with CI/CD for automated environment monitoring
📈 Add Business Metrics specific to your use cases
🌐 Scale to Multiple Projects and regions

Resources & Links

📚 Full Documentation: GitHub Repository
🛠️ Issues & Support: GitHub Issues
📖 GCP Pub/Sub Docs: Official Documentation
🎯 AppDynamics Extensions: Machine Agent Extensions

Conclusion

Monitoring GCP Pub/Sub doesn't have to be complicated. With the right automation and tooling, you can have comprehensive visibility into your messaging infrastructure in under 30 minutes.

The solution we've built together provides enterprise-grade monitoring with minimal operational overhead. Your teams can now:

Detect issues proactively before they impact users
Understand performance patterns across your messaging layer
Make data-driven decisions about scaling and optimization
Maintain high availability with real-time health monitoring

Have questions or want to share your success story? Drop a comment below or reach out on GitHub!

⭐ If this guide helped you, please give the GitHub repository a star and share it with your team!

Are you a member of the Splunk Community?

Complete Guide: Monitor Google Cloud Pub/Sub with Splunk AppDynamics Machine Agent