If you’ve enabled Automatic Discovery in your install of the Splunk Distribution of the OpenTelemetry Collector by following along with Part 2 of this series, then you’re already sitting back, relaxing, and watching it automatically detect and monitor services like your PostgreSQL databases. But what about more complex scenarios? What if you’re running multi-tenant databases where each tenant has its own schema? Or managing a microservice architecture with dozens of interconnected services? What about monitoring an entire application stack – databases, caches, web servers, and message queues. How do you monitor all of these things without manually configuring each component? Automatic Discovery.
In this final part of this Automatic Discovery series, we’ll explore practical real-world use cases that go beyond basic setup. You’ll see how to use discovery rules, conditional configuration, and labeling to handle complex production scenarios, all while maintaining the hands-off automation that makes Automatic Discovery so powerful.
Note: Automatic Discovery supports specific infrastructure services – primarily databases, caches, and select message queues. The exact list of supported services depends on which receivers are bundled in your Collector version, and not every receiver bundled with the Collector supports Automatic Discovery. For the full list of Automatic Discovery third-party application supported in Kubernetes and Linux, check out the documentation on Automatic Discovery of Apps and Services.
Database Observability at Scale
Databases are perfect candidates for Automatic Discovery because they follow predictable patterns that often require environment-specific configuration. If you followed the setup in Part 2, you’ve already seen basic PostgreSQL discovery in action. Now let’s explore more advanced database monitoring scenarios.
The Splunk Distribution of the OpenTelemetry Collector includes bundled receiver configurations for common infrastructure services. These work automatically once you enable discovery.enabled:true in your Helm values. The examples in this section show how to customize those bundled configurations for specialized scenarios like advanced metric collection or environment-specific settings. For more standard database deployments, the bundled configurations handle discovery automatically without any additional configuration beyond providing credentials.
Advanced PostgreSQL Discovery with Custom Metrics
If you want to customize which PostgreSQL metrics are collected or you need to monitor specific databases based on pod labels, here’s how to extend the bundled PostgreSQL receiver configuration in your Helm values:
This configuration extends the default PostgreSQL discovery to:
- Enable TLS verification for secure connections
- Collect specific database performance metrics beyond the defaults
- Use environment-based credentials for security
- Monitor all discovered PostgreSQL instances
Note: We’re illustrating common PostgreSQL receiver capabilities, and the actual metric names, configuration options, and available settings may differ depending on your receiver version. Collection intervals also depend on your receiver’s capabilities and Collector version. Reducing or increasing collection intervals will impact the volume of data sent to Splunk Observability Cloud, which can impact your metric billing and cardinality.
MySQL/MariaDB Discovery
MySQL and MariaDB work similarly to PostgreSQL – the bundled configurations handle basic discovery automatically. Here’s how to configure credentials in your Helm values:
The Collector will automatically discovery any MySQL of MariaDB pods in your cluster and start collecting metrics using these credentials.
Note: don’t forget to create the mysql-monitoring secret in your cluster before deploying, just like you did for PostgreSQL in Part 2 of this series:
Monitoring Multiple Services in Your Application Stack
Real applications often consist of multiple interconnected services – databases, caches, message queues, web servers. Let’s say you’re running a typical web application with:
- PostgreSQL database for persistent data
- Redis cache for session storage
- MongoDB for document storage
- The application itself
Here’s what a complete Helm values would look like in order to enable discovery for all of them:
Deploy this configuration once, and the Collector will automatically discover and monitor every PostgreSQL, Redis, and MongoDB instance that appears in your cluster.
Handling Production versus Non-Production Environments
One common question that often comes up around Automatic Discovery is, “Can I use the same discovery configuration in development and production?” The short answer: yes, with minor adjustments.
One of the benefits of Automatic Discovery is that the same configuration works across all environments, however, you’ll generally want different collection intervals or different credentials depending on the environment.
The recommended approach is to use the same Helm values configuration across environments but create environment-specific secrets. For example, in development, your secrets might look like:
And in production it might look like:
To adjust collection intervals with environment-specific values, simply update the collection_interval value in each environment-specific values file:
- values-production.yaml:
- values-development.yaml:
Then deploy the appropriate values file:
Beyond Databases: Other Supported Services
Automatic Discovery isn’t limited to databases. The Splunk Distribution of the OpenTelemetry Collector includes bundled configurations for other common infrastructure services like:
- Databases:
- PostgreSQL
- MySQL/MariaDB
- MongoDB
- Redis
- Message queues:
- RabbitMQ
- Kafka
- Web servers:
- NGINX
- HTTP Server
Here’s a basic example of Helm values for Redis discovery:
Once your Redis instance is deployed, you’ll see metrics start flowing into Splunk Observability Cloud.
Next Steps
Automatic Discovery reduces manual work, decreases the potential for observability gaps, and eliminates lag time between deploying infrastructure and gaining visibility into it. If you’re ready to get started with Automatic Discovery, here’s a checklist summary of the steps to ensure deployment goes smoothly:
The beauty of Automatic Discovery is that once it’s enabled, every new service you deploy gets monitored automatically. Mo more extensive manual configuration. No more observability gaps.
Don’t yet have Splunk Observability Cloud? Try it free for 14 days!
Resources
