Is there a way to configure token passthrough in Splunk HEC exporter for OpenTelemetry collector?

[stephen]

Hello team,

I'm testing a use case where I have two OpenTelemetry collectors (with Splunk HEC exporter configured) running on two different hosts. I want to send logs from one collector (collector A) to the other (collector B) and then send it to Splunk using the Splunk HEC exporter. Is there a way to configure the HEC token in collector A and passthrough the token to collector B so that collector B can send logs from collector A to Splunk? Thank you.

[richgalloway]

What problem are you trying to solve by using an intermediate forwarder?  Why can't both collectors send directly to HEC?

[stephen]

This is for having multiple different collectors, each running on different hosts. These collectors will receive traces, metrics, and logs from applications running on their respective hosts. Then they'll each send data (traces, metrics, and logs) to a central collector, which will export them to 3rd party vendors (Splunk being one of them). This use case allows each "local" collectors to have minimum configuration. All they need to do is to forward data to the central collector and the central collector will handle exporting to 3rd party vendors. I saw that the Signalfx exporter has a configuration option for access_token_passthrough, which is used to preserve datapoint origin. I wanted to know if there is something similar for the Splunk HEC exporter. If not, I wanted to ask if the Splunk team has any plans to implement this feature. 

[richgalloway]

Interesting.  If I understand the scenario correctly, only the central collector will need a HEC token since it is the only one connecting to HEC.  Tokens authenticate connections, not data.

If you still need separate tokens then perhaps the collectors can be made to exchange them.

If you'd like to ask Splunk to implement pass-through HEC tokens then go to https://ideas.splunk.com

BTW, HEC is not an "exporter".  HEC *collects* (that's what the 'C' stands for) so it's more of an "importer".

[stephen]

Yes, the central collector will need a HEC token to export to Splunk but it will not be configured at the central collector's config file. Each local collector will have a different HEC token configured in their respective config files and it will forward the logs (along with the HEC token) using Splunk HEC exporter to the central collector. This is where a token passthrough option would be used, it will be configured at the local collector's config file. Then the central collector will use the received token to authenticate connection to Splunk and export the received data.

I apologize for the confusion. I used the term Splunk HEC exporter because it is the name of the exporter in the Otel repository (https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/splunkhecexport...).  Thank you for the link. I'll make a post there.

[roryab]

Fixed here:

https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/5435