Troubleshooting SplunkHecExporter in opentelemetry-collector-contrib

[bschaap]

I need advice on troubleshooting SplunkHecExporter.  I'm using an OpenTelemetry Collector to accept logs via OTLP, export them to an on-prem Splunk Heavy Forwarder, which them forwards them to Splunk Cloud.  Below is my configuration.  I'm sending some test logs from Postman but the logs don't arrive in Splunk Cloud.  I see the arrival of the logs in the OpenTelemetry Collector through the debug exporter.  I confirmed connectivity to the Splunk Heavy Forwarder by setting an invalid token which results in an authentication error.  Using a valid token doesn't result in any debug logs being recorded.  Any suggestions on troubleshooting?

exporters:
  debug:
    verbosity: normal
  splunk_hec:
    token: "<valid token>"
    endpoint: "https://splunkheavyforwarder.mydomain.local:8088/services/collector/event"
    source: "oteltest"
    sourcetype: "oteltest"
    index: "<valid index>"
    tls:
      ca_file: "/etc/otel/config/certs/ca_bundle.cer"
    telemetry:
      enabled: true
    health_check_enabled: true
    heartbeat:
      interval: 10s
service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: []
      exporters: [splunk_hec, debug]
    telemetry:
      logs:
        level: "debug"

[Av2]

@bschaap I'm also facing the same issue. I used that sample OTLP spec log json file. How did you fix it?

[bschaap]

If I recall it worked when I sent test logs from my client app which is instrumented with the Faro Web SDK library.  I didn't go back to compare the log contents against the otlp spec log to figure out the difference since it was working.  I think it was silently failing for some reason.

[bschaap]

I reviewed the _internal index and discovered that the heartbeat records to the _internal index (the SplunkHECExporter code shows this too).  My HEC token doesn't allow that index and was erroring. 

This explains why the heartbeat wasn't working but I'm still unable to determine why my otlp logs aren't making it to Splunk via the exporter.  I reviewed the other _internal logs but am unable to find anything.

For additional context, I'm reusing an existing Splunk Heavy Forwarder that has many logs going through it.  This is the first time I've used the SplunkHecExporter exporter in the OtelCollector.  The debug logs show that splunk_hec is registered to export logs.

[richgalloway]

Confirm you see the HF's internal logs in Splunk Cloud (search for "index=_internal host=<<your HF name>>").  If you don't then the HF is not connecting to Splunk Cloud (did you install the Universal Forwarder app on the HF?) and that should be fixed first.  If the HF's logs are in the cloud then use them to determine why otel data is not getting in.

[bschaap]

@richgalloway   I discovered that sending the log to the raw endpoint works.  However, the log is stripped down to the log body which is what the documentation states (ie. from docs - `send only the log's body`).  

export_raw: false

endpoint: ".../services/collector/raw"

Resulting log (only the log body shows up in Splunk) -

   Example log record

I'm still having trouble figuring out why logs aren't coming through the normal event endpoint.  The sample log I'm using is from the otlp log data spec.

[richgalloway]

Splunk HEC will not accept events on the /event endpoint if they are not valid JSON.  I believe an error will be returned in that case, however.

[bschaap]

Ended up getting it to work using a different sample log from the app that will be submitting the OTLP logs.  For some reason the example in the otlp spec wouldn't work on the /event endpoint using Postman but would on the /raw endpoint.  Though with the /raw endpoint it was just the log body and not all the context around it.  Thanks for your help!