Automatic Discovery Part 1: What is Automatic Discovery in Splunk Observability Cloud and Why It Matters

[CaitlinHalla]

If you’ve ever deployed a new database cluster, spun up a caching layer, or added a load balancer, you know it can involve a lot of manual configuration, YAML wrangling, and crossing your fingers that you didn’t miss anything critical.

Automatic Discovery in the Splunk Distribution of the OpenTelemetry Collector helps eliminate some of this toil so you can automatically and easily begin monitoring your infrastructure and applications in Splunk Observability Cloud the moment they come online.

Traditional Observability Setup Challenges

Setting up observability for modern infrastructure can be a lift. You deploy a new database instance, and suddenly you’re writing Collector configurations, setting up service discovery, and manually defining which metrics to collect. Scale that across dozens of services, multiple environments, constant deployments, and you’ve got a recipe for potential observability gaps.

With traditional approaches, you need to:

• Manually configure monitoring for each new service
• Update Collector configs every time infrastructure changes
• Maintain separate configurations for different environments
• Hope that your teammates remember to add monitoring when they deploy a new service

And the result of all this? Potential blind spots in your observability coverage and time lost to configuration rather than innovation and problem solving.

Automatic Discovery to the Rescue

Automatic Discovery thankfully flips this model on its head. Instead of manually configuring monitoring for every service, you enable Automatic Discovery once, and the Splunk Distribution of the OpenTelemetry Collector automatically detects, configures, and starts monitoring new services as they appear.

Automatic Discovery detects and collects signal data from third-party services such as databases and web servers by automatically generating configuration snippets that you can modify and incorporate into your existing configuration. Think of it as a smart assistant that constantly scans your infrastructure, recognizes common services, and automatically sets up monitoring based on what it finds.

How Automatic Discovery Works

With discovery mode enabled, the Collector performs intelligent detection through a multi-step process.

Observer Extensions Scan Your Environment 

Automatic Discovery uses observer extensions – like the k8s_observer in Kubernetes or the host_observer in Linux – to continuously watch for infrastructure endpoints. These observers poll at short intervals and report discovered endpoints (pods, services, processes) to the Collector typically within seconds of them becoming available.

Preflight Testing with Bundled Configurations 

The Splunk Distribution of the OpenTelemetry Collector includes bundled receiver configurations for common infrastructure services like PostgreSQL, Redis, Kafka, NGINX, etc. During a "preflight" discovery phase, the Collector tests these bundled configurations against the endpoints discovered by the observers to see which ones successfully connect and collect data. This means that newly deployed services typically begin reporting metrics within moments of starting up.

Automatic Integration of Successful Discoveries 

When a receiver configuration successfully discovers a service (for example, detecting a database on a specific port), the Collector automatically incorporates that receiver into its active configuration. Metrics start flowing to Splunk Observability Cloud immediately – no Collector restarts or manual configuration updates required. 

Discovery State and Persistence

Discovery state is maintained in memory by the Collector during runtime. When a service is discovered and monitoring begins, the Collector tracks that endpoint as long as it remains active. If the Collector restarts, it performs a fresh discovery scan of the current environment state at startup, re-evaluating all known observer sources like nodes, pods, or network interfaces. This means it will rediscover any services that are still running but won’t retain information about services that stopped while the Collector was down.

For ephemeral services, like containers or scaled-down pods, the Collector stops attempting to collect metrics from those endpoints. In Splunk Observability Cloud, stopped services typically drop from the active discovery list after the next few polling cycles (usually within minutes).

With everything up and running, you can verify what’s been discovered by checking the Collector logs for discovery events or by viewing the new services and metrics that appear automatically in Splunk Observability Cloud’s Infrastructure Navigator.

Real-World Impact

If you were to deploy a new cache cluster with traditional manual configuration, before you could get any metrics you would need to:

• Update the Collector configuration with receiver settings
• Specify connection details
• Restart the Collector
• Verify metrics are flowing

If you happen to be running multiple cache instances across different namespaces, you’d need to repeat this process for each one.

With Automatic Discovery enabled, you deploy your new cache cluster, and the Collector immediately detects the new pods and automatically begins collecting metrics. Within seconds, you see cache-specific metrics – no manual Collector configuration, no YAML editing, no hoping someone remembers to add monitoring.

This same pattern applies for all your supported infrastructure services. Deploy it, and it’s automatically monitored.

The Benefits of Automatic Discovery

Eliminates Observability gaps

With traditional manual configuration, it’s easy to miss new services or forget to update monitoring when services change.

Reduces Time to Visibility

Instead of waiting for someone to manually configure monitoring, new services become observable the moment they start running. This is critical for fast-moving teams practicing continuous deployment.

Scales Effortlessly

Whether you’re managing 10 services or 1,000, Automatic Discovery scales with your infrastructure. Define your rules once, and they apply consistently across all environments.

Reduces Configuration Drift

Manual configurations tend to drift across environments and teams. Automatic Discovery ensures consistent monitoring configuration based on your defined rules, reducing environment-specific quirks.

Enables Self-Service Observability

Development teams can deploy new services with proper monitoring automatically – no observability team intervention required.

Where Automatic Discovery Fits in Your Observability Strategy

Automatic discovery isn’t meant to replace thoughtful observability design – it’s meant to eliminate the tedious manual work that prevents you from focusing on what matters. Here’s how it fits into the bigger picture:

• Discovery layer: Automatic Discovery handles the detection and basic configuration of monitoring for standard services and patterns.
• Customization layer: for services that need specialized monitoring or custom metrics, you still have full control over Collector configuration.
• Alerting layer: once Automatic Discovery sets up basic monitoring, you can layer on custom alerting rules, SLOs, and business-specific metrics using Splunk Observability Cloud’s full feature set.

Think of Automatic Discovery as providing the observability foundation that lets you focus on higher-value observability practices like building meaningful dashboards, setting up intelligent alerting, and creating SLIs that matter to your business.

Getting Started: What’s Next?

Automatic Discovery shines brightest in dynamic environments where services are constantly being deployed, scaled, and updated. If you’re running Kubernetes workloads, containerized applications, or cloud-native infrastructure, Automatic Discovery can drastically simplify your observability operations.

The setup process is straightforward, but there are important considerations around security, configuration management, and best practices that can make the difference between a smooth rollout and a frustrating experience.

In Part 2 of this series, we’ll walk through the step-by-step process of enabling Automatic Discovery, including how to securely handle credentials, avoid common pitfalls, and configure discovery rules that work for specific infrastructure patterns.

Ready to stop manually configuring observability for every new service? You can explore Automatic Discovery by setting it up with Splunk Observability Cloud – try it free for 14 days and see how it simplifies your setup.

Resources

• Automatic discovery of apps and services
• Getting started with the Splunk Distribution of the OpenTelemetry Collector
• Exploring the OpenTelemetry Collector’s Kubernetes annotation-based discovery