Need Help Resolving OpenTelemetry Collector Data Reception Issue in Dockerized Django Application

Anders — Thu, 14 Mar 2024 05:10:45 GMT

Hi Splunk Community,

I'm working on a Django-based website server running inside a Docker container, and I'm facing an issue with OpenTelemetry Collector (Otel) data reception. Despite following the official Splunk documentation for installing Otel within a Docker container. I'm encountering an issue where the Otel installed on my VM isn't receiving any data from my Django application.

Here are the warning logs from the Otel container:

2024-03-14 04:49:05,592 WARNING [opentelemetry.exporter.otlp.proto.grpc.exporter] [exporter.py:293] [trace_id=0 span_id=0 resource.service.name=website trace_sampled=False] - Transient error StatusCode.UNAVAILABLE encountered while exporting metrics to localhost:4317, retrying in 32s.

Initially, my Dockerfile was configured with OTEL_EXPORTER_OTLP_ENDPOINT='localhost:4317'. Considering that might be the issue, I updated it to OTEL_EXPORTER_OTLP_ENDPOINT='otelcol:4317', aiming to directly communicate with the Otel collector service running as a Docker container. However, I'm still observing attempts to connect to localhost:4317 in the error logs.

Here's a brief overview of my setup:

Django application running in a Docker container.
OpenTelemetry Collector deployed as a separate Docker container named 'otel-collector'.
Dockerfile for the Django application updated to use the OpenTelemetry Collector container endpoint.
Could anyone provide insights or suggestions on what might be going wrong here? How can I ensure that my Django application correctly sends telemetry data to the Otel Collector?
Thank you in advance for your help and suggestions!

Re: Need Help Resolving OpenTelemetry Collector Data Reception Issue in Dockerized Django Application

bishida — Thu, 14 Mar 2024 19:32:15 GMT

Hi,

Here are a couple of thoughts to consider. If you want to run the OTel collector in its own container, are you using appropriate networking mode for your container? For example, port 4317 (as well as others) would need to bind to the host networking ports so that your other container running your application can refer to "localhost:4317". You may want to start the OTel collector container first and then try some simple tests on the host command line to make sure it's accessible (e.g., 'telnet localhost 4317' or 'nc -vz localhost 4317') https://lantern.splunk.com/Data_Descriptors/Docker/Setting_up_the_OpenTelemetry_Demo_in_Docker

Also, I didn't see any mention of container orchestration (kubernetes). This might be a good use-case for kubernetes where you run your application container with kubernetes and the OTel collector can run in a side-car configuration. https://docs.splunk.com/observability/en/gdi/opentelemetry/collector-kubernetes/kubernetes-config.html

topic Need Help Resolving OpenTelemetry Collector Data Reception Issue in Dockerized Django Application in Splunk Observability Cloud

Need Help Resolving OpenTelemetry Collector Data Reception Issue in Dockerized Django Application

Re: Need Help Resolving OpenTelemetry Collector Data Reception Issue in Dockerized Django Application