Open Telemetry Errors in Windows

[Dominic32]

Getting a ton of these Telemetry errors in Event Log of a windows server with at UF installed. They started a few days ago. What could be causing them? No changes have been made to the UF or splunk infrastructure recently.

1.6987038408387303e+09 error exporterhelper/queued_retry.go:183 Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "name": "signalfx", "error": "Permanent error: \"HTTP/2.0 401 Unauthorized\\r\\nContent-Length: 0\\r\\nDate: Mon, 30 Oct 2023 22:10:40 GMT\\r\\nServer: istio-envoy\\r\\nWww-Authenticate: Basic realm=\\\"Splunk\\\"\\r\\nX-Envoy-Upstream-Service-Time: 5\\r\\n\\r\\n\"", "dropped_items": 50}

go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send

/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.53.0/exporter/exporterhelper/queued_retry.go:183

go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send

/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.53.0/exporter/exporterhelper/metrics.go:132

go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1

/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.53.0/exporter/exporterhelper/queued_retry_inmemory.go:119

go.opentelemetry.io/collector/exporter/exporterhelper/internal.consumerFunc.consume

/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.53.0/exporter/exporterhelper/internal/bounded_memory_queue.go:82

go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func2

/builds/o11y-gdi/splunk-otel-collector-releaser/.go/pkg/mod/go.opentelemetry.io/collector@v0.53.0/exporter/exporterhelper/internal/bounded_memory_queue.go:69

[Dominic32 · Accepted solution]

Error was being caused by the OpenTelemetry Collector service, which is separate from the Splunk UF service. I stopped the service and errors are no longer being logged in Event Viewer.