You might have seen Splunk’s recent announcement about donating the OpenTelemetry Injector to the community.  If you haven’t, it’s worth a read. It shares how Splunk and Dash0 partnered to make OpenTelemetry easier to adopt across every environment. In this post, I’ll unpack how the Injector works, how it differs from the OpenTelemetry Operator and Automatic Discovery, and why it’s a big step forward for enterprises looking to extend observability across all their workloads, not just the containerized ones. The Enterprise Reality: Hybrid Environments Are Here to Stay If you’re like many large IT organizations, your environment spans cloud-native applications, virtual machines, and on-prem workloads that make perfect sense to keep where they are. Some systems are decades old, others are brand new but together, they make up the backbone of your business. And that’s where observability can get complicated. Different environments often mean different monitoring approaches  which can lead to increased tool costs, inconsistent methods for measuring system health, and what I like to call “swivel-chair analysis”  switching between dashboards and data sources to piece together what’s really happening. The OpenTelemetry Injector helps close that gap by extending OpenTelemetry’s reach across all your environments, putting you in control of your data. It brings the same consistency, scalability, and flexibility across on-prem, VM-based, and hybrid workloads that you expect from your cloud-native applications and turns that consistency into deeper, more contextual insights that drive faster analysis and better understanding of system behavior. Enter the OpenTelemetry Injector Originally developed by Splunk and rewritten in Zig by Dash0 for the OpenTelemetry project, the Injector provides zero-touch auto-instrumentation for host-based workloads. It’s designed for environments that can’t (or don’t need to) be containerized  the types of applications running on VMs, bare-metal servers, or traditional tiered architectures. Instead of editing application code, rebuilding containers, or writing custom startup logic, the Injector works directly at the Linux host level, automatically attaching the right OpenTelemetry agents when your applications start up. Here’s what that looks like: Installed as a lightweight shared library: libotelinject.so Written in Zig for cross-platform compatibility across glibc and musl Enabled by preloading the library with LD_PRELOAD or configuring it globally in /etc/ld.so.preload Automatically injects environment variables and agent paths for Java, Node.js, and .NET Configured via a single file: /etc/opentelemetry/otelinject.conf That means no code changes, no complex pipelines, and no manual rebuilds, just consistent telemetry from applications wherever they run. How It Works The diagram below shows how the Injector integrates into the Linux dynamic linking process to automatically load instrumentation before the application starts running: When a process starts, the operating system links the program and loads its system libraries. If the OpenTelemetry Injector is configured via LD_PRELOAD or /etc/ld.so.preload, it’s loaded first allowing it to set environment variables and point supported runtimes (Java, Node.js, .NET) to their respective OpenTelemetry agents. The application then executes normally, but with telemetry automatically emitted to your OpenTelemetry Collector.  Note: The command below is an example of globally enabling the Injector: echo /usr/lib/opentelemetry/libotelinject.so | sudo tee -a /etc/ld.so.preload Always confirm the correct library path and configuration in the official OpenTelemetry Injector documentation, as installation paths may vary across Linux distributions. Alternative Activation via systemd In addition to the preload method, the OpenTelemetry Injector also supports systemd-based activation as noted in recent community discussions and blog posts (see Isabella Langan’s article on the OTel website) This approach is especially useful for administrators who want finer control or operate in environments where global preloads aren’t allowed. Using a standard systemd drop-in configuration, you can set environment variables that activate the Injector for specific managed services. You can configure a systemd drop-in like this: sudo systemctl edit myapp.service Then add: [Service] Environment="LD_PRELOAD=/usr/lib/opentelemetry/libotelinject.so" After saving, restart the service: sudo systemctl restart myapp.service The next time the application starts, the Injector loads automatically, attaches the appropriate language agent, and begins sending telemetry  all without modifying the original service configuration If You’re Like Me, You Might Be Thinking… “Wait, isn't this basically what the OpenTelemetry Operator does?” It’s a fair question, and one I asked myself when I first started learning about the Injector. Both approaches automate instrumentation, but they work in different layers of the stack. Aspect Injector (Host-Level) Operator Auto-Instrumentation (Kubernetes) Auto-Discovery (Collector Runtime) Primary Purpose Injects environment variables and language agents into host-based processes at startup. Adds auto-instrumentation agents to Kubernetes pods automatically. Dynamically detects workloads/endpoints and configures receivers on the fly. Where It Runs Linux hosts or containers (LD_PRELOAD / /etc/ld.so.preload) Kubernetes admission webhook Inside the OTel Collector What It Does Configures app runtime env and agent hooks (Java, Node.js, .NET) Adds agents + env vars via annotations (inject-java: true) Finds and scrapes telemetry endpoints dynamically Target Use Case VM-based services, traditional app tiers, hybrid environments Cloud-native environments with centralized consistency Dynamic services that appear/disappear quickly Control Surface /etc/opentelemetry/otelinject.conf Pod annotations + Instrumentation CRD Collector discovery / observers config Goal Summary Make the app emit telemetry. Make the cluster emit telemetry consistently. Find and collect telemetry dynamically.   So in summary: Connecting the Dots: Injector + Automatic Discovery If you haven’t already, check out Caitlin Halla’s three-part blog series on Automatic Discovery in the Splunk Distribution of the OpenTelemetry Collector. Automatic Discovery simplifies observability by automatically detecting and monitoring services (like databases, caches, and web servers) as they come online with no YAML wrangling, no restarts, and no manual receiver configuration. Together, the Injector and Automatic Discovery close two sides of the same loop: The Injector ensures your applications emit telemetry automatically. Automatic Discovery ensures your collectors find and capture it automatically. That combination means faster time to value and seamless observability across the entire technology stack. Why This Matters: Extending Observability Everywhere We’ve already nailed comprehensive visibility across Kubernetes and microservices. But enterprises are still hybrid. There are workloads that make sense to stay on hosts or in multi-tier architectures. The Injector is a big step forward in making those environments first-class citizens of observability. This contribution from Splunk and Dash0 helps organizations: Extend OpenTelemetry coverage to every environment Reduce operational overhead by removing per-app instrumentation work Standardize data collection through a unified OTel pipeline Integrate on-prem telemetry with cloud-native observability practices It’s about leverage  bringing the power of OpenTelemetry everywhere your applications live. Try It Out If you’re managing hosts, VMs, or hybrid workloads, the Injector can simplify your rollout dramatically. Grab it from the OpenTelemetry Injector repository, follow the quick-start guide, and connect it to your existing OpenTelemetry Collector. If you’re already using Splunk Observability Cloud, you can send that data directly  no additional configuration required. Wrap-Up OpenTelemetry has already changed how we think about observability. With the addition of the Injector, it’s now possible to extend that simplicity to every environment  from Kubernetes clusters to traditional servers. Observability shouldn’t stop at the edge of your cluster. The OpenTelemetry Injector ensures it doesn’t. Check out the Splunk announcement if you haven’t yet, and keep an eye on the OpenTelemetry project for ongoing updates. We’re just getting started. ... View more

As a Developer Evangelist at Splunk, my team and I are constantly tinkering with technology to explore its nuances and discover practical use cases. Most of that experimentation is short-lived, helping to drive awareness through demos and content. But occasionally, a solution sticks and lives on well beyond the initial project. A great example is splunkgames.com—a simple single-page app that acts as a portal to interactive experiences we’ve built to help grow awareness of OpenTelemetry, Observability concepts, and Splunk’s Observability portfolio. You’ll even find links to our blogs and YouTube playlists there (if you haven’t checked it out yet, go take a look!). While we’re not running a battle-tested SRE team for these environments (I know, I know… don’t judge!), we do need to keep the site available and generally responsive. And you guessed it—we leverage SplunkSynthetic Monitoring, part of Splunk Observability Cloud,to keep tabs on the health of the landing page and the availability of the interactive content. Recently, with Splunk joining the Cisco party, we’ve also gained access to another great collaboration tool: Webex. Since our team primarily communicates through Webex Messaging, I figured—why not integrate Splunk Observability Cloud alerts directly into the channels we already use? Like many things in tech, I found scattered pieces of the puzzle and stitched them together. A huge shoutout to the ThousandEyes Webex Integration Guide, which—although focused on ThousandEyes—walks through exactly how to configure the Webex side of the integration (and yes, I literally copied-pasta’ed steps 1-5). Having worked on the Splunk Observability Cloud webhook configurations before—including partnering with the team on the “Splunk Platform” integration—and recalling Caitlin’s excellent blog on sending Observability alerts to Jira, I decided to spend a couple of hours wiring it all together. The result? A relatively clean integration that drops alerts directly into our Webex Space. My team can see alerts as they fire, chat about them directly in the thread with things like “got it,” “looking into it,” or my personal favorite—“Mike! Not again!” Quick Template Overview: What This Solution Provides Leverages Webex Adaptive Cards for a clean, structured alert view. Includes a fallback to a standard text message if Adaptive Cards aren’t supported. Uses Splunk Observability Cloud's custom webhook “if” logic to display optional fields like the detector’s Runbook URL and TIP only when they’re defined in the detector configuration. Dynamically sets the message’s color using the severityDecoder helper function, aligning severity levels to appropriate visual cues. (See Splunk Webhook Helper Functions for details.) Adds an actionable button that drives directly to Splunk Observability Cloud’s alert view—helping expedite troubleshooting right from the message. When it’s all said and done (correctly) the result looks something like this Step by Step: Leveraging Cisco Webex for Alert Notifications This article shows you how to receive Splunk Observability Cloud alert notifications in a Cisco Webex App space using webhooks. Create a new Webex bot: A Webex bot lets you integrate services into Webex. In the Webex App, create a new bot to receive alert notifications from the ThousandEyes webhook. For detailed instructions, see Creating a Webex Bot. Use the screenshots below to help you create your bot. 2. Copy the bot's access token: After setting up your new Webex bot, copy the bot's access token and save it somewhere handy. You'll need this token when you configure the Splunk Observability cloud  integration (step 6) 3. In Webex App, create a space to receive alert notifications from Splunk Observability Cloud: Skip this step if you already have a Webex App space where you'd like to receive alerts. 4. Add the Webex App bot to the space: In this example, the bot named "Splunk Observability Cloud Alerts" is added to the "Splunk O11y Cloud Example Alert Space" Webex App space.. Then Click “Create” 5. Copy the space's roomId: You'll use the roomId with the bot's access token (from Step 2) to configure the ThousandEyes integration. Now that the Webex bot is a member of your Webex App space, look up the roomId of your Webex App space and copy it. To find the roomId: Use your Webex App login to access the Webex API site. Using a web browser, navigate to the following URL and click “Run” to look up the roomId: https://developer.webex.com/docs/api/v1/rooms/list-rooms 6. Create A new WebHook integration in Splunk Observability cloud. Log into your Observability cloud Organization. Choose “Data Management” from the left nav. Then Click “Add Integration” Use the Search box and locate the “WebHook” notification services. Click the integration tile to start the setup: 7. Configure the Webhook            Click Next at the Webhook summary page Enter the following information into the “Configure connection” form, then click “Next” Field Value Name Provide a descriptive name. This name will appear in the “Alert recipients” section of your detector configuration URL https://webexapis.com/v1/messages Method POST Shared Secret Leave default Headers Key  Value Content-type application/json Authorization Bearer [token_from_step_2]   Pro-tips: Keys and Values may be case sensitive In the Authorization header the value should be in the following format “Bearer[space][token_value_from_step2]” (without the quotes) For example; if your token value from step2 was “123456789ABCDEFG” then the value of your Authorization header would be “Bearer 123456789ABCDEFG” (without the quotes) 8. Copy and Paste the template into the “Customize message” payload. Update line 6 with the roomID obtained in Step5 (replace “<YOUR_WEBEX_ROOM_ID>” with your token) then click Next Custom Payload (copy and paste - remember to remove the angle brackets around your Webex room ID but leave the quotes) {{!-- Webex Room ID: Update this with the target Webex Room ID where notifications should be sent. Retrieve the Room ID using the Webex API or Webex Control Hub. --}} { "roomId": "<YOUR_WEBEX_ROOM_ID>", {{!-- Markdown Fallback: Used if Adaptive Cards aren’t supported by the Webex client. Provides a readable plain-text version of the alert details. Best practice is to update this section as well as the adaptive card --}} "markdown": "<strong><u>{{{encodeString messageTitle}}}</u></strong>\n<strong>Severity:</strong> {{severity}}\n<strong>Status:</strong> {{{statusExtended}}}\n<strong>Source of Alert: </strong>{{{coalesce src 'src not found'}}}\n<strong>Alert Description:</strong>\n{{{messageBody}}}\n<strong>Detector: </strong>[{{{encodeString detector}}}]({{{detectorUrl}}}&orgID={{orgId}})\n{{#if runbookUrl}} [Runbook URL]({{runbookUrl}})\n{{/if}}{{#if tip}} <strong>Detector TIP specified:</strong>\n{{{encodeString tip}}}\n{{/if}}", "attachments": [ { "contentType": "application/vnd.microsoft.card.adaptive", "content": { "$schema": "http://adaptivecards.io/schemas/adaptive-card.json", "type": "AdaptiveCard", "version": "1.3", "body": [ {{!-- Alert Title with Dynamic Severity Color. The severityDecoder helper function is used to determine the color based on severity. For more details, see Splunk Webhook Helper Functions: https://dev.splunk.com/observability/docs/integrations/webhook_integration_overview#Helper-functions --}} { "type": "TextBlock", "text": "{{{encodeString messageTitle}}}", "wrap": true, "weight": "bolder", "size": "large", "color": "{{{severityDecoder Critical='attention' Major='warning' Minor='accent' Warning='accent' Info='good' ok='good' default='default'}}}" }, {{!-- Generic Alert Fields: Includes event timestamp, severity, status, source, and description --}} { "type": "TextBlock", "text": "Event Raised at: {{timestamp}}", "wrap": true }, { "type": "TextBlock", "text": "Severity: {{severity}}", "wrap": true }, { "type": "TextBlock", "text": "Status: {{{statusExtended}}}", "wrap": true }, { "type": "TextBlock", "text": "Source of Alert: {{{coalesce src 'src not found'}}}", "wrap": true }, { "type": "TextBlock", "text": "Alert Description:\n{{{messageBody}}}", "wrap": true }, {{!-- Optional TIP Section: Included only if 'tip' is present in the detector config--}} {{#if tip}} { "type": "TextBlock", "text": "Detector TIP specified:\n{{{encodeString tip}}}", "wrap": true }, {{/if}} {{!-- Action Buttons: Provide quick access to investigate the alert or view the runbook (if one is defined in the detecor config) --}} { "type": "ActionSet", "actions": [ { "type": "Action.OpenUrl", "title": "Investigate in Splunk Observability Cloud", "url": "{{{detectorUrl}}}&orgID={{orgId}}" } {{#if runbookUrl}}, { "type": "Action.OpenUrl", "title": "Runbook", "url": "{{{runbookUrl}}}" } {{/if}} ] } ] } } ] } Click “Save” at the “Review and save” page 9. Update your Detectors “Alert Receipts” Select “Detectors & SLOs”, then select “Detectors” , identify the detectors you’d like to receive Webex notifications about and select “Manage subscriptions” Click the “Add recipient" button, choose “Webhook” from the menu, choose your newly created webhook from the drop down, then click the “save” button 10. Once a Detector alerts, you should see an Adaptive card appear in your newly created space Wrapping Things Up: Already using Splunk Observability Cloud and Webex Messaging? Why not take it a step further and bring alerts directly into the conversations you’re already having? If you’re not using either product, why not? Check out Webex and/or start your Splunk Observability Cloud Trial today! Leverage the template above as-is or customize it to meet your team’s specific needs. It’s a simple way to improve visibility, reduce response times, and (let’s be honest) have a little fun roasting your teammates when things go sideways. Leave a comment below if you use this or found it helpful and stay tuned for more. ... View more

To start, if you're new to synthetic monitoring, I recommend exploring this synthetic monitoring overview. In today's fast-paced world of web development, browser synthetic testing is a vital tool in the observability toolbox. With technical debt piling up faster than code reviews, it's essential to have a strategy in place that gives you confidence in your application's health. As someone who’s been responsible for delivering high-quality, high-confidence observability services across several Fortune 500 companies, I can confidently say: I slept better knowing our critical client-facing applications had synthetic monitoring coverage.  Synthetic monitoring isn’t just a buzzword; it’s an active safety net that ensures your site’s availability, performance, and functionality are up to par, even when no one’s watching. When your synthetic browser monitoring spots a problem, especially on mission-critical apps, you know it’s a real problem that needs to get fixed fast. As an observability practitioner, I've spent countless hours in the trenches, implementing & tuning synthetic browser monitoring to catch potential issues early. These tools work in harmony with passive monitoring approaches like RUM and APM. Over time, I’ve gathered insights and best practices that I’d like to share with you—especially if you’re looking to fine-tune your synthetic browser tests for success. 1. Power in Simplicity   Enterprise web applications are stuffed with features, but not all of them are equally important. For an e-commerce site, the critical functions include the landing page, login, product search, and payment. While features like live chat, profile customization, and social sharing are cool, they won’t break the business if they’re down for a while. Draft a Short User Journey Start by working with someone who knows the ins and outs of the app—maybe the end-user or product manager. Together, map out the user journey they typically take, noting prerequisites (like login before checkout) and success criteria (e.g., seeing “Welcome, John Doe” after logging in).  If possible, record this session to reference while building your synthetic tests. You’ll also want to group certain steps into transactions—this helps you logically organize the journey, generate transactional KPI’s, and focus on key points in the flow. Using transactions lets you focus on the relevant things the user does, not the minutiae of what page was loaded when. Focus on Critical Actions When designing your tests, stay focused. It’s tempting to test everything, but remember: just because you can doesn’t mean you should. Stick to the critical actions you identified earlier, like logging in, searching, and making purchases.  Finally, focus on underlying service invocation —you want your synthetic tests to complement passive monitoring strategies like APM, not replace functional testing. Ideally, your synthetic testing will also exercise key backend components. You’d do this by making sure to build user workflows that reflect all the critical functionality your site has to offer. As a bonus, in Splunk Synthetic Monitoring, you can even see directly how user transactions affect your backend services from each synthetics run. 2. Validate User Actions A synthetic test is only as good as its validation. Ensuring each user action produces the expected result is critical. Confirm Actions Have the Intended Result Leverage assertions to validate that content renders correctly. For instance, testing a search function at Splunk T-shirt co? Instead of searching for “Supernova limited Edition T-Shirt,” invoke a search for “shirt”. In this example, when the limited edition shirt is no longer available the test would fail and need to be updated, however it utilizing the search term “shirt” would always return results.  Build Robust Tests to Avoid False Positives False positives are the bane of synthetic tests. A robust strategy can minimize them, such as using fuzzy searches instead of exact terms, like mentioned above. You can also implement assertions to wait for elements to load before interacting with them, preventing a race condition from failing your test. Also, be mindful of things like maintenance/downtime windows, as tests run during these times may result in false positives. And, of course, test in the same environment configuration that your users will experience—this includes matching viewport size, cookies, and browser settings. 3. Test Hygiene Matters Keeping your tests organized is just as important as the tests themselves. Follow a Naming Convention A consistent naming convention makes it easier to manage your tests. Use names that clearly reflect the application, environment, or action being tested. For instance: `Application XYZ Checkout_Success_Prod_Test`. Add Custom Properties (Tags) Tagging tests with custom properties—like `app_name`, `environment` , support_level, or `component`—helps with test isolation and troubleshooting. You can use these tags to quickly filter your tests in order to isolate issues or you might leverage them in your alerts to provide additional context to responders. Align these tags with your organization's existing tagging standards for maximum clarity. Use Consistent Testing Locations Ensure your tests are run from consistent locations to get reliable, comparable results. Make sure those locations are aligned with your users’ geography—whether internal or external. Leverage variables  Many robust Synthetic monitoring solutions provide a means to variablize parameters utilized in your test configuration. Leveraging variables allows for centralized management of variables. A common use case would be to store a regularly leveraged username and password in variables. In this example, if you need to update the username or password, you now simply update the variables instead of searching through your synthetics test configuration and updating the tests one-by-one.     4. Act! Alert! Alert! Once your tests are in place, you need a reliable alerting strategy. High-confidence alerting is crucial to respond quickly when an issue arises. The biggest advantage of synthetic monitoring is the knowledge that a failure is generally highly-actionable (assuming you’ve followed the rules earlier in this article to generate good tests.) Build Detectors for KPI Thresholds Set up detectors to alert on KPI thresholds, such as availability and performance. For example: Start by monitoring on page availability, server errors (HTTP status codes > 500), or deviations of the KPI for your synthetic browser test transactions (I.E. search is slow to return results).  Start Simple with Availability Alerts Begin with basic availability alerts to ensure your site is online. Some common availability alerts include checks for SSL certificate validity and server connection errors (status codes > 500). Once your tests are stable, evolve your alerting thresholds to react to seasonal or historical anomalies for critical performance metrics, rather than relying on static thresholds. A great place to start with performance alerts is at the transaction level, for example, generating alerts if the time it takes to render search results or authenticate a user deviates from the historic baseline.  And remember: document everything! Knowing what your test does and why will help your team respond appropriately when alerts trigger. Analyze and Optimize Synthetic browser tests are capable of generating vast amounts of data. Common synthetics metrics include page performance timings, page web vitals (CLS, LCP, TBT), page Connection timings, page resource and error counts, score metrics (such as Lighthouse), page content size metrics, and transaction-level metrics (duration, requests, & total size) This data can be extremely valuable for troubleshooting failures and/or identifying optimization opportunities . For example:  Regularly review your Core Web Vitals and determine if they differ from industry standards. Implement optimizations/improvements and leverage the data to determine the effect. Integrate your deployment pipelines with your synthetic testing data. This will allow you to overlay application deployments and quickly correlate changes with availability issues and/or deviations to performance metrics. Develop common synthetics test dashboards that help analyze your synthetic data over time. Reference these dashboards in your alerts, as this analysis should expedite the response process and ultimately reduce MTTR.   Conclusion Synthetic browser tests are an invaluable part of the modern observability stack, but only if you approach them with care and strategy. Keep your focus on critical user journeys, validate thoroughly, maintain solid test hygiene, and always stay alert (literally). By applying these tips, you’ll not only enhance your synthetic testing framework but also provide more robust observability coverage for your applications. If you’re interested in content similar to this, I’d encourage you to check out the Observability Developer Evangelist blogs and/or our YouTube Playlist.    ... View more