Facing issues in Implementing Auto Instrumentation method level tracing with Splunk.OpenTelemetry.Autoinstrumentation

[santoshboorlaga]

We are trying to implement method-level tracing using the `splunk.opentelemetry.autoinstrumentation` package (version 1.9.0) in a .NET Core Web API application targeting .NET 9. We’ve set up the code, but we’re facing runtime issues. Despite trying various solutions, including reinstalling the `splunk.opentelemetry.autoinstrumentation` package, the issues persist. Could you please help us resolve these and suggest any necessary modifications? Do we need to add any collector?

Another issue : When we set "CORECLR_ENABLE_PROFILING": "0" then we are able to see traceids in console but unable to see the traceid in splunk APM window.

Error 1:

System.ExecutionEngineException
HResult=0x80131506
Source=<Cannot evaluate the exception source>
StackTrace:
<Cannot evaluate the exception stack trace>

at OpenTelemetry.AutoInstrumentation.NativeMethods+Windows.AddInstrumentations(System.String, OpenTelemetry.AutoInstrumentation.NativeCallTargetDefinition[], Int32)
at OpenTelemetry.AutoInstrumentation.NativeMethods.AddInstrumentations(System.String, OpenTelemetry.AutoInstrumentation.NativeCallTargetDefinition[])
at OpenTelemetry.AutoInstrumentation.Instrumentation.RegisterBytecodeInstrumentations(Payload)
at OpenTelemetry.AutoInstrumentation.Instrumentation.Initialize()
at DynamicClass.InvokeStub_Instrumentation.Initialize(System.Object, System.Object, IntPtr*)
at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(System.Object, System.Reflection.BindingFlags)
at System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
at System.Reflection.MethodBase.Invoke(System.Object, System.Object[])
at OpenTelemetry.AutoInstrumentation.Loader.Loader.TryLoadManagedAssembly()
at OpenTelemetry.AutoInstrumentation.Loader.Loader..cctor()
at OpenTelemetry.AutoInstrumentation.Loader.Loader..ctor()
at System.RuntimeType.CreateInstanceDefaultCtor(Boolean, Boolean)
at System.RuntimeType.CreateInstanceImpl(System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
at System.Reflection.Assembly.CreateInstance(System.String)
at StartupHook.Initialize()
at System.StartupHookProvider.ProcessStartupHooks(System.String)

Error 2:

Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

at OpenTelemetry.AutoInstrumentation.NativeMethods+Windows.AddInstrumentations(System.String, OpenTelemetry.AutoInstrumentation.NativeCallTargetDefinition[], Int32)
at OpenTelemetry.AutoInstrumentation.Instrumentation.RegisterBytecodeInstrumentations(Payload)
at OpenTelemetry.AutoInstrumentation.Instrumentation.Initialize()
at OpenTelemetry.AutoInstrumentation.Loader.Loader.TryLoadManagedAssembly()
at OpenTelemetry.AutoInstrumentation.Loader.Loader..cctor()
at OpenTelemetry.AutoInstrumentation.Loader.Loader..ctor()
at System.RuntimeType.CreateInstanceDefaultCtor(Boolean, Boolean)
at System.Reflection.Assembly.CreateInstance(System.String)
at StartupHook.Initialize()
at System.StartupHookProvider.ProcessStartupHooks()

launchSettings.json:

{
"$schema": "https://json.schemastore.org/launchsettings.json",
"iisSettings": {
"windowsAuthentication": false,
"anonymousAuthentication": true,
"iisExpress": {
"applicationUrl": "http://localhost:47665",
"sslPort": 44339
}
},
"profiles": {
"WebApplication3": {
"commandName": "Project",
"dotnetRunMessages": true,
"launchBrowser": true,
"launchUrl": "swagger",
"applicationUrl": "https://localhost:7146;http://localhost:5293",
"environmentVariables": {
"ASPNETCORE_ENVIRONMENT": "Development",
"OTEL_SERVICE_NAME": "MyDotNet6WebApi",
"OTEL_EXPORTER_OTLP_ENDPOINT": "https://ingest.XX.signalfx.com/v2/trace/otlp",
"OTEL_EXPORTER_OTLP_HEADERS": "X-SF-Token=fdsfsdfsd-M-fdsfsdfsd-r",
"OTEL_DOTNET_AUTO_ENABLED": "true",
"OTEL_DOTNET_TRACES_METHODS_INCLUDE": "WebApplication3.Controllers.SampleController.DoSomething",
"OTEL_DOTNET_AUTO_TRACES_ADDITIONAL_SOURCES": "Microsoft.AspNetCore.Http,System.Net.Http",
"OTEL_TRACES_EXPORTER": "otlp,console",
"OTEL_EXPORTER_OTLP_PROTOCOL": "http/protobuf",
"OTEL_DOTNET_AUTO_INSTRUMENTATION_LOGS": "true",
"OTEL_DOTNET_AUTO_INSTRUMENTATION_ENABLED": "true",
"CORECLR_ENABLE_PROFILING": "1",
"OTEL_DOTNET_AUTO_LOG_DIRECTORY": "C:\\temp\\otel-logs",
"CORECLR_PROFILER": "{918728DD-259F-4A6A-AC2C-4F76DA9F3EAB}",
"DOTNET_STARTUP_HOOKS": "%USERPROFILE%\\.nuget\\packages\\opentelemetry.autoinstrumentation.startuphook\\1.10.0\\lib\\netcoreapp3.1\\OpenTelemetry.AutoInstrumentation.StartupHook.dll",
"OTEL_DOTNET_AUTO_HOME": "%USERPROFILE%\\.nuget\\packages\\splunk.opentelemetry.autoinstrumentation\\1.9.0",
"CORECLR_PROFILER_PATH_64": "%USERPROFILE%\\.nuget\\packages\\opentelemetry.autoinstrumentation.runtime.native\\1.10.0\\runtimes\\win-x64\\native\\OpenTelemetry.AutoInstrumentation.Native.dll",
"CORECLR_PROFILER_PATH_32": "%USERPROFILE%\\.nuget\\packages\\opentelemetry.autoinstrumentation.runtime.native\\1.10.0\\runtimes\\win-x86\\native\\OpenTelemetry.AutoInstrumentation.Native.dll"
}
}
}
}

SampleController.cs:

using Microsoft.AspNetCore.Mvc;
using System.Diagnostics;
using System.Net.Http;

namespace WebApplication3.Controllers
{
[ApiController]
[Route("api/[controller]")]
public class SampleController : ControllerBase
{
private readonly IHttpClientFactory _httpClientFactory;

public SampleController(IHttpClientFactory httpClientFactory)
{
_httpClientFactory = httpClientFactory;
}

[HttpGet("execute")]
public async Task<IActionResult> Execute()
{
await DoSomething();
var traceId = Activity.Current?.TraceId.ToString();
return Ok(new { message = "Request executed", traceId });
}

[NonAction]
public async Task DoSomething()
{
using var activity = new Activity("SampleController.DoSomething").Start();

// Internal HTTP call to another endpoint
var response = await _httpClientFactory.CreateClient().GetAsync("https://jsonplaceholder.typicode.com/todos/1");
var content = await response.Content.ReadAsStringAsync();
Console.WriteLine(content);
}
}
}

rapid response for Splunk Splunk Add-On for OpenTelemetry Collector 

[bishida]

Hi,
Quick sanity check: are you certain there's nothing being instrumented that uses an older version of .NET? The reason I ask is because the service name might be indicating .NET 6? The 1.9 instrumentation won't work with .NET6 or .NET7--you'd have to try downgrading to 1.8.

"OTEL_SERVICE_NAME": "MyDotNet6WebApi"

[santoshboorlaga]

@livehybrid Thanks for the quick reply 😊. 

We are trying to avoid manual instrumentation in the code completely. I am trying to do a onetime setup so that whatever the methods that we want to trace, I can add them through env variable OTEL_DOTNET_TRACES_METHODS_INCLUDE.

Please suggest me if there is any other better approach where I can implement method level tracing in a better way to implement.

Regarding the error

All the paths mentioned are correct and the dlls exists in those paths. We have checked the versions and all the versions exists in .nuget package. Could you please help me in resolving the issue. 

"%USERPROFILE%\\.nuget\\packages\\opentelemetry.autoinstrumentation.startuphook\\1.10.0\\lib\\netcoreapp3.1\\OpenTelemetry.AutoInstrumentation.StartupHook.dll",
"OTEL_DOTNET_AUTO_HOME": "%USERPROFILE%\\.nuget\\packages\\splunk.opentelemetry.autoinstrumentation\\1.9.0",
"CORECLR_PROFILER_PATH_64": "%USERPROFILE%\\.nuget\\packages\\opentelemetry.autoinstrumentation.runtime.native\\1.10.0\\runtimes\\win-x64\\native\\OpenTelemetry.AutoInstrumentation.Native.dll",
"CORECLR_PROFILER_PATH_32": "%USERPROFILE%\\.nuget\\packages\\opentelemetry.autoinstrumentation.runtime.native\\1.10.0\\runtimes\\win-x86\\native\\OpenTelemetry.AutoInstrumentation.Native.dll"

[livehybrid]

Hi @santoshboorlaga 

The error  AccessViolationException and ExecutionEngineException suggest problems with the native profiler component.

• Check Environment Variables - Update your launchSettings.json to ensure all paths and versions match.
• Ensure these NuGet packages are installed with matching versions
• Modify Your Controller for Better Tracing:

using Microsoft.AspNetCore.Mvc;
using System.Diagnostics;
using System.Net.Http;
using System.Diagnostics.Metrics;
using System.Collections.Generic;

namespace WebApplication3.Controllers
{
    [ApiController]
    [Route("api/[controller]")]
    public class SampleController : ControllerBase
    {
        private readonly IHttpClientFactory _httpClientFactory;
        private static readonly ActivitySource _activitySource = new ActivitySource("WebApplication3.Controllers");

        public SampleController(IHttpClientFactory httpClientFactory)
        {
            _httpClientFactory = httpClientFactory;
        }

        [HttpGet("execute")]
        public async Task<IActionResult> Execute()
        {
            // Create a parent activity for the entire request
            using var requestActivity = _activitySource.StartActivity("Execute", ActivityKind.Server);
            requestActivity?.SetTag("http.method", "GET");
            requestActivity?.SetTag("http.path", "/api/sample/execute");
            
            await DoSomething();
            
            // Get the current trace ID
            var traceId = Activity.Current?.TraceId.ToString() ?? requestActivity?.TraceId.ToString();
            
            requestActivity?.SetTag("request.success", true);
            return Ok(new { message = "Request executed", traceId });
        }

        [NonAction]
        public async Task DoSomething()
        {
            // This method should be instrumented by auto-instrumentation
            // But we can also add explicit instrumentation for redundancy
            using var activity = _activitySource.StartActivity(
                "DoSomething",
                ActivityKind.Internal,
                parentContext: Activity.Current?.Context ?? default);
            
            activity?.SetTag("operation.type", "data_fetch");
            
            try
            {
                // Internal HTTP call to another endpoint
                var client = _httpClientFactory.CreateClient();
                var response = await client.GetAsync("https://jsonplaceholder.typicode.com/todos/1");
                response.EnsureSuccessStatusCode();
                
                var content = await response.Content.ReadAsStringAsync();
                activity?.SetTag("operation.success", true);
                Console.WriteLine($"TraceId: {activity?.TraceId}");
                Console.WriteLine(content);
            }
            catch (Exception ex)
            {
                activity?.SetTag("operation.success", false);
                activity?.SetTag("error.message", ex.Message);
                activity?.SetStatus(ActivityStatusCode.Error);
                throw;
            }
        }
    }
}

• Add Program.cs Configuration - You need to manually add OpenTelemetry services to your Program.cs file for better control and to ensure proper integration:

Regarding Your CORECLR_ENABLE_PROFILING Issue, you mentioned that when you set "CORECLR_ENABLE_PROFILING": "0", you see trace IDs in the console but not in Splunk APM. This makes sense because:

• Setting this to 0 disables the profiler-based auto-instrumentation
• Your explicit instrumentation still works (via Activity objects)
• But without the profiler, the method-level tracing won't work

The solution is to keep CORECLR_ENABLE_PROFILING set to 1 but fix the underlying issues:

1. Make sure all paths in environment variables are correct
2. Check OpenTelemetry configuration as shown above

For production environments, using an OpenTelemetry Collector is recommended:

• It acts as a buffer between your application and Splunk
• Provides retry mechanisms for temporary connection issues
• Can batch telemetry data for more efficient transmission
• Reduces the overhead on your application

For development, you can continue using direct export to Splunk as you're doing now, but for production, I'd recommend setting up a collector.

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will