My role as an Observability Specialist at Splunk provides me with the opportunity to work with customers of all sizes as they implement OpenTelemetry in their organizations.  If you read my earlier article, 3 Things I Love About OpenTelemetry, you'll know that I'm a huge fan of OpenTelemetry.  But like any technology, there's always room for improvement.  In this article, I'll share three areas where I think OpenTelemetry could be improved to make it even better.    #1:  Make OpenTelemetry Even Easier  While OpenTelemetry has come a long way in the past few years, making it even easier would allow more organizations to adopt it, and result in a faster time to value for everyone.  I’ll share a few specific examples below.    Expand Auto-Instrumentation Coverage One example where ease of use could be improved is in the instrumentation of languages that don’t support auto-instrumentation, such as Golang.   The good news is that efforts are already underway to build a solution that provides auto-instrumentation for Go using eBPF.  While this is still a work in progress, you can learn more about it in GitHub and even try it out yourself (on a non-production app, please!)     Troubleshooting Guidance Many practitioners that I work with find it challenging to troubleshoot OpenTelemetry-related issues.  While some of these issues are with the OpenTelemetry collector, the majority are at the instrumentation level. For example, they may not see the spans they’re expecting.  Or perhaps the application even crashes on startup when auto-instrumentation is added.  Regardless of the specific instrumentation issue, there’s frequently confusion about where to start troubleshooting.  More often than not, the focus is on the OpenTelemetry Collector logs, due to a lack of understanding of where the instrumentation occurs vs. what role the collector plays.   I believe the OpenTelemetry community as a whole would benefit from further guidance on troubleshooting techniques.  This could take the form of expanded documentation, videos demonstrating the troubleshooting process for real-world issues, or tutorials that let practitioners go through these processes themselves with a mock application.  Generative AI can also play a role in providing troubleshooting guidance.  For example, as of July 2024, ChatGPT is already able to help with collector configuration tasks and provide direction on how to troubleshoot OpenTelemetry-related issues.  For example, if we provide the following prompt:  I'm having trouble with the OpenTelemetry collector.  Can you help me to troubleshoot this error?             warn kubelet/accumulator.go:102 failed to fetch container metrics {"kind": "receiver", "name": "kubeletstats", "data_type": "metrics", "pod": "mypod-5888f4d9fb-lbbww", "container": "mypod", "error": "failed to set extra labels from metadata: pod \"a3473219-4ab1-427c-b0f2-226a6e5271e5\" with container \"mypod\" has an empty containerID"}           ChatGPT was able to dissect the error message and provide the following interpretation:  “The key part of the error message is has an empty containerID. This suggests that the OpenTelemetry collector is attempting to fetch metrics for a container within a pod (mypod-5888f4d9fb-lbbww), but it cannot proceed because the container ID is empty.” It also provided suggested troubleshooting steps such as confirming that the Kubernetes API Server is up and running, and ensuring the container has a valid Container ID associated with it.  While it’s not perfect, Generative AI is already helpful for troubleshooting OpenTelemetry issues today, and it will only continue to get better in the future.    #2:  Narrow the Gap with Traditional APM Agents One of my favorite aspects of traditional APM solutions that rely on proprietary APM agents is their ability to apply code-level instrumentation without requiring code changes.  It would be great to see similar “no-code change required” capabilities added to OpenTelemetry.  I’ve provided a few examples below.    Creating Spans It’s sometimes helpful to capture spans that go above and beyond what auto-instrumentation provides.  With OpenTelemetry today, this typically requires making a code change.  And while the code change itself is straightforward, with a few additional lines of code at most, it can take time to get this prioritized in a team’s sprint, tested, and pushed out to production.  OpenTelemetry does provide some support for this today with Java.  Specifically, with Java it’s possible to create additional spans by adding a system property to the JVM at startup.  See Creating spans around methods with otel.instrumentation.methods.include for further details.   It would be great to see similar capabilities added to languages beyond Java.    Capturing Span Attributes As an observability enthusiast, I believe it’s critical to capture span attributes to ensure engineers have the context they need during troubleshooting.  For example, let’s say we have a user profile service, and one of the endpoints is /get-profile, which retrieves the profile of a particular user.  We may find that the response time of this service varies widely.  Sometimes it responds in a few milliseconds, and other times it takes upwards of 1-2 seconds.  Adding span attributes to provide context about the request, such as the user ID and the number of items in that user’s history, is critical to ensure the engineer has the information they need for troubleshooting.  The /get-profile operation might run slowly for users that have a large number of items in their history, but it wouldn’t be possible to determine this without having those attributes included with the trace.  While some traditional APM solutions provide similar capabilities without code changes, capturing span attributes with OpenTelemetry currently requires code changes.  As with creating spans, these code changes aren’t difficult.  But it can be challenging to prioritize these types of changes amongst competing feature requests and get them into production in a timely manner.  There is some support in OpenTelemetry today for capturing HTTP headers as span attributes with the Java agent.   It would be great to see OpenTelemetry extend these capabilities, and add further support for capturing span attributes without requiring code changes.    Profiling Spans that are captured with OpenTelemetry’s auto-instrumentation will tell us how long calls between services are taking, and whether any errors are occurring.  This can be supplemented with manually created spans to provide insight into long-running tasks that require a deeper level of visibility.  But when it comes to finding the exact method or line of code in your code that’s causing an application to run slowly, we need to move beyond spans and look at profiling instead. Many traditional APM tools provide some form of profiling, but this level of detail hasn't been available with OpenTelemetry.  The good news is that the process of adding profiling to OpenTelemetry is already underway.  In fact, OpenTelemetry announced upcoming support for profiling in March 2024.  Please see OpenTelemetry announces support for profiling for details.  Note:  while profiling is in the process of being added to OpenTelemetry, Splunk distributions of OpenTelemetry already include AlwaysOn Profiling capabilities.    #3:  Expand Scope to Network and Security Domains OpenTelemetry provides a wealth of information about applications and the infrastructure they run on.  This includes apps running on traditional host-based environments as well as containerized apps running on Kubernetes.  It also includes observability data from other components which applications depend on, such as databases, caches, and message queues.  This data goes a long way in determining why an application is performing slowly, or why the error rate for a particular service has suddenly spiked.  But sometimes, issues go beyond the application code and server infrastructure that the applications run on.  It would be helpful to see OpenTelemetry broaden its scope and provide visibility into additional domains.     Network We’ve all heard the joke about software engineers blaming the network whenever something goes wrong in their application.  Well, the truth is that sometimes the problem *is* caused by the network.  Yet engineers responsible for building and maintaining these applications rarely have direct insights into how the network is performing.  So for more holistic visibility into anything that could be impacting application performance, it would be wonderful to see OpenTelemetry add support for the network domain in the future.  This could include ingesting metric and log data from existing network monitoring solutions, or pulling data directly from network devices themselves.     Security While it’s important for an application to be available and performant, none of that matters if the application isn’t secure.   Since OpenTelemetry already has a wealth of information about the applications it instruments, having OpenTelemetry expand into the security domain would open up a whole new set of use cases for observability data.    For example, OpenTelemetry could gather information about what specific packages and versions are used by an instrumented application.  This could take the form of a new “security” signal, with a corresponding set of security-related semantic conventions that ensure this data is captured in a consistent manner across different languages and runtimes.    The security signals could then be analyzed by an observability backend, and engineers could be alerted when a security vulnerability is present in their application.  And by correlating security-related signals with existing signals such as traces and the upcoming profiling signal, observability backends could also determine when a particular vulnerability is being exercised  by analyzing which code paths are actively being exercised.    Summary Thanks for taking the time to hear my thoughts on OpenTelemetry.  Please leave a comment or reach out to let us know how you would make OpenTelemetry even better.  ... View more

In 2022, I made the decision to focus my career on OpenTelemetry.  I was excited by the technology and, after working with proprietary APM agent technology for nearly a decade, I believed that it was the future of instrumentation.  This ultimately led me to join Splunk in 2023 as an Observability Specialist.  Splunk Observability Cloud is OpenTelemetry-native, so this role allowed me to work extensively with OpenTelemetry as customers of all sizes implemented it in their organizations.  So how am I feeling about OpenTelemetry in 2024?  Well, I’m even more excited about it than before!  In this article, I’ll share the top three things that I love about OpenTelemetry.      #1:  It’s Easy to Use!  In the beginning, instrumenting an application with OpenTelemetry required code changes.  This is referred to as code-based instrumentation or manual instrumentation.   It was great for early enthusiasts who were passionate about observability, wanted full control over their telemetry, and didn’t mind spending time instrumenting their code by hand.   OpenTelemetry has come a long way since then, with some form of auto-instrumentation support for the most popular languages such as Java, .NET, Node.js, and Python.  This is also referred to as zero-code solutions.  This makes it easier for organizations to get up and running quickly with OpenTelemetry, just like they would with a proprietary solution based on traditional APM-agents, while also giving them the flexibility to layer on custom instrumentation if desired.  The advent of the OpenTelemetry Operator for Kubernetes has also made it easier to instrument applications running in Kubernetes.  Specifically, the OpenTelemetry Operator can automatically inject and configure instrumentation libraries for multiple languages.  This makes it simple for organizations using Kubernetes to instrument their applications using OpenTelemetry.  Ultimately, these ease of use improvements have made OpenTelemetry more accessible, and have dramatically reduced the time to value, as it’s now possible to be up and running with OpenTelemetry in just minutes.    #2:  Flexibility of the OpenTelemetry Collector Architecture  The collector is my favorite part of OpenTelemetry (and perhaps the most flexible yet elegant architecture I’ve encountered in my career).  While the concept of the collector originated in 2017 as part of the OpenCensus project at Google, with OpenTelemetry it has evolved to become a mature and highly flexible software component that many organizations depend on.  The OpenTelemetry Collector allows one or more pipelines to be configured, which define how data is received, processed, and exported.  This data can include metrics, traces, and logs.  A pipeline can be depicted as follows:   Source:  https://opentelemetry.io/docs/collector/architecture/  Each pipeline can have one or more Receivers that receive metric, trace, or log data from various sources.  The data in the pipeline then passes through one or more Processors, which can transform, filter, or manipulate the data in other desired ways.  Finally, the data is then exported using Exporters to one or more observability backends, which allows organizations to decide exactly where they want to send their observability data.  This architecture provides near infinite flexibility.  For example, if you want to send your metrics to one observability backend and your traces and logs to another, no problem!  Or if you want to send a subset of traces to a backend that resides in a particular jurisdiction, to comply with data residency requirements, sure!   Here are a few additional examples of what you can do with the collector:  Use the Resource Detection Processor to gather additional information about the host it’s running on, and add it as context to metrics, spans, and logs.  Use the Redaction Processor to redact sensitive data before it leaves your network.   Use the Transform Processor to rename span attributes, to ensure naming conventions are enforced across all of your observability data.  The architecture also allows collectors to be chained.  Typically, this means running a collector in agent mode on each host or Kubernetes node to gather data from applications running on that host along with infrastructure-related data.  These collectors will then export their data to another collector running in gateway mode, which will perform additional processing on the data before it’s exported to one or more observability backends.  This collector architecture is depicted in the following diagram:  Source:  https://opentelemetry.io/docs/collector/architecture/    #3:  Support for Logs   While metrics and traces have been Generally Available (GA) in OpenTelemetry for several years, it wasn’t until November 2023 that logs joined these other signals and became GA as well.   This was a tremendous step forward, as logs play a critical role in the troubleshooting process, frequently providing the details that engineers need to understand why a particular issue is occurring.  I love that OpenTelemetry provides so many different ways to ingest logs, including support for Fluent Bit and Fluentd with the Fluent Forward Receiver.  Or the versatile Filelog Receiver, which can be configured to ingest logs from just about any file-based source.  And it gets even better with Kubernetes, which now includes a Logs Collection Preset.  This preset, which is available in the collector Helm chart, uses the Filelog Receiver under the hood, and provides all of the configuration needed to automatically collect logs using the standard output from Kubernetes containers.  Collecting logs with OpenTelemetry means that we can apply all of the power and flexibility of the collector that was discussed in the previous section to logs.  And the metrics, traces, and logs that are collected with OpenTelemetry share the same Semantic Conventions, which makes it possible to correlate these different signals together.   For example, with Semantic Conventions, log events that include the TraceId, SpanId, and TraceFlags fields can now be linked to the corresponding trace data.  This makes it easy to jump between related logs, traces, and metric data when troubleshooting an issue, where time is of the essence.  I also love that some languages such as Java have started to collect logs automatically with OpenTelemetry.  So there’s no need to even use the Filelog Receiver to ingest the application logs, as everything is captured using OpenTelemetry SDKs under the hood.  In addition to requiring less configuration effort, collecting logs in this manner is also more performant, as there’s no need to read application log files from the host filesystem and parse them.    Summary Thanks for taking the time to hear my thoughts on OpenTelemetry.  Please leave a comment or reach out to let us know what you love about OpenTelemetry.    ... View more