Setup for sending distributed traces to Splunk APM

[_pravin]

Hi All,

I am trying to send distributed traces to Splunk APM, but I cannot send directly to the observability cloud, as this application is hosted on the internet.

To give a better picture: We have developers working on any organisation's server, and they want to send the traces from this server to Splunk APM, and also should be converted to protobuf format before being sent, but the data should always go through the 'intermediate Splunk instance'  as we have clearance only for this server to send data to Splunk APM.

The development server currently uses the OTeL collector to send traces to a local Jaeger instance, but we want to use Splunk APM in the future.

I have created a diagram to illustrate how it looks.

Can someone suggest to me or guide me on how to achieve this setup?

Thanks in advance.

Pravin

[bishida]

Hi,

Happy to help try to break this down.

In your first point, you mention that an application is hosted on the Internet and you’re unable to send traces directly to Observability Cloud. Normally, you can do exactly that. You should be able to configure your application instrumentation to send directly to Observability Cloud by setting a few env vars or javaagent flags. (e.g., SPLUNK_ACCESS_TOKEN, OTEL_EXPORTER_OTLP_TRACES_PROTOCOL, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT)

Example, here is docs for how to do that with a Java app.
https://help.splunk.com/en/splunk-observability-cloud/manage-data/instrument-back-end-services/instr...

It looks like your second point may be describing some on-prem servers or applications where you want to use an “intermediate server”. You can use a Splunk OpenTelemetry collector gateway for this. You can configure the gateway to send metrics and traces to Observability Cloud and export your logs to your Splunk Enterprise/Cloud/Heavy Forwarder.
https://help.splunk.com/en/splunk-observability-cloud/manage-data/splunk-distribution-of-the-opentel...

[_pravin]

Hi @bishida ,

Thanks for the inputs.

I checked the documentation and have updated the diagram to show what should be happening under the hood.

The setups on the left are part of the organisation's infrastructure, and on the right are exposed to the internet.

Is this feasible to implement?

Thanks,

Pravin

[_pravin]

Hi @livehybrid ,

Currently, I am using the intermediary Splunk for sending data to Splunk Cloud (again on the internet) via HEC. Now, in addition to this existing setup, I want to use this intermediary Splunk to send data to Splunk APM as well, using another HEC token, but the data has to be converted to protobuf format first before sending to APM.

Thanks,

Pravin

[livehybrid]

Hi @_pravin 

Are you using the Intermediary Splunk server with HEC/Splunk forwarding for something else in addition to Otel feeds? It seems to me like the most appropriate architecture would be to use a Gateway Otel Collector that sends to Splunk APM?

For more info I'd recommend checking out https://help.splunk.com/en/splunk-observability-cloud/manage-data/splunk-distribution-of-the-opentel...

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing