How to connect OpenTelemetry Java agent to Splunk Collector in Docker?

[thatusername]

I use the OpenTelemetry Java agent to monitor FusionAuth in one Docker container, and send the output to the Splunk OpenTelemetry Docker container in Gateway mode.

Here's a diagram of my system architecture:

```mermaid
graph LR
subgraph I[Your server]
direction LR
subgraph G[Docker]
H[(Postgresql)]
end
subgraph C[Docker]
direction BT
D(OpenTelemetry for Java) --> A(FusionAuth)
end
subgraph E[Docker]
B(Splunk OpenTelemetry collector)
end
end
C --> G
C --> B
E --> F(Splunk web server)
style I fill:#111
```

The Splunk container runs correctly and exports sample data to Splunk Observability Cloud. I can see it in the dashboard.

FusionAuth and the Java agent run correctly.

But the Otel sender cannot send to the Otel collector. I get network errors:

```sh

| [otel.javaagent 2024-06-07 13:52:40:936 +0000] [OkHttp http://otel:4317/...] ERROR io.opentelemetry.exporter.internal.http.HttpExporter - Failed to export logs. The request could not be executed. Full error message: Connection reset
fa | java.net.SocketException: Connection reset
fa | at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:328)
fa | at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:355)

...

[otel.javaagent 2024-06-07 13:52:42:847 +0000] [OkHttp http://otel:4317/...] ERROR io.opentelemetry.exporter.internal.http.HttpExporter - Failed to export spans. The request could not be executed. Full error message: Connection reset by peer
fa | java.net.SocketException: Connection reset by peer
fa | at java.base/sun.nio.ch.NioSocketImpl.implWrite(NioSocketImpl.java:425)
fa | at java.base/sun.nio.ch.NioSocketImpl.write(NioSocketImpl.java:445)
fa | at java.base/sun.nio.ch.NioSocketImpl$2.write(NioSocketImpl.java:831)
fa | at java.base/java.net.Socket$SocketOutputStream.write(Socket.java:1035)

```

I'm using the standard configuration file for Splunk Linux Collector - https://github.com/signalfx/splunk-otel-collector/blob/main/cmd/otelcol/config/collector/otlp_config...

Below is my docker compose file

```yaml

services:
db:
image: postgres:latest
container_name: fa_db
ports:
- "5432:5432"
environment:
PGDATA: /var/lib/postgresql/data/pgdata
POSTGRES_USER: ${POSTGRES_USER}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
healthcheck:
test: [ "CMD-SHELL", "pg_isready -U postgres" ]
interval: 5s
timeout: 5s
retries: 5
networks:
- db_net
volumes:
- db_data:/var/lib/postgresql/data

fa:
# image: fusionauth/fusionauth-app:latest
image: faimage
container_name: fa
# command: "tail -f /dev/null"
depends_on:
db:
condition: service_healthy
environment:
DATABASE_URL: jdbc:postgresql://db:5432/fusionauth
DATABASE_ROOT_USERNAME: ${POSTGRES_USER}
DATABASE_ROOT_PASSWORD: ${POSTGRES_PASSWORD}
DATABASE_USERNAME: ${DATABASE_USERNAME}
DATABASE_PASSWORD: ${DATABASE_PASSWORD}
FUSIONAUTH_APP_MEMORY: ${FUSIONAUTH_APP_MEMORY}
FUSIONAUTH_APP_RUNTIME_MODE: ${FUSIONAUTH_APP_RUNTIME_MODE}
FUSIONAUTH_APP_URL: http://fusionauth:9011
SEARCH_TYPE: database
FUSIONAUTH_APP_KICKSTART_FILE: ${FUSIONAUTH_APP_KICKSTART_FILE}
networks:
- db_net
ports:
- 9011:9011
volumes:
- fusionauth_config:/usr/local/fusionauth/config
- ./kickstart:/usr/local/fusionauth/kickstart
extra_hosts:
- "host.docker.internal:host-gateway"

otel:
image: quay.io/signalfx/splunk-otel-collector:latest
container_name: fa_otel
environment:
SPLUNK_ACCESS_TOKEN: "secret"
SPLUNK_REALM: "us1"
SPLUNK_LISTEN_INTERFACE: "0.0.0.0"
SPLUNK_MEMORY_LIMIT_MIB: "1000"
SPLUNK_CONFIG: /config.yaml
volumes:
- ./config.yaml:/config.yaml
networks:
- db_net
# no host ports are needed as communication is inside the docker network
# ports:
# - "13133:13133"
# - "14250:14250"
# - "14268:14268"
# - "4317:4317"
# - "6060:6060"
# - "7276:7276"
# - "8888:8888"
# - "9080:9080"
# - "9411:9411"
# - "9943:9943"

networks:
db_net:
driver: bridge

volumes:
db_data:
fusionauth_config:

```

The FusionAuth Dockerfile starts FusionAuth like this:

```sh

exec "${JAVA_HOME}/bin/java" -javaagent:/usr/local/fusionauth/otel.jar -Dotel.resource.attributes=service.name=fusionauth -Dotel.traces.exporter=otlp -Dotel.exporter.otlp.endpoint=http://otel:4317 -cp "${CLASSPATH}" ${JAVA_OPTS} io.fusionauth.app.FusionAuthMain <&- >> "${LOG_DIR}/fusionauth-app.log" 2>&1

```

Why can't the FusionAuth container connect to http://otel:4317 please?