Signalfx AlwaysOn Profiling

[nohadarwish]

I'm trying to enable SignalFx AlwaysOn Profiling for my Java application. The app is already instrumented to:

• Send metrics directly to the ingest endpoint, and

• Send traces via a Collector agent running on the host.

  I have a couple of questions:

  1. Can the ingest endpoint also be used for profiling, similar to how it's used for metrics?
     If yes, could you please share the exact endpoint format or a link to the relevant documentation?

  2. I attempted to enable profiling by pointing to the same Collector endpoint used for tracing. The logs indicate that the profiler is enabled, but I’m also seeing a message saying "Exporter failed", without a specific reason for the failure.

     Could you help me troubleshoot this issue?

     Here are the relevant log entries:

     com.splunk.opentelemetry.profiler.ConfigurationLogger - -----------------------
     com.splunk.opentelemetry.profiler.ConfigurationLogger - Profiler configuration:
     com.splunk.opentelemetry.profiler.ConfigurationLogger -                  splunk.profiler.enabled : true
     com.splunk.opentelemetry.profiler.ConfigurationLogger -                splunk.profiler.directory : /tmp
     com.splunk.opentelemetry.profiler.ConfigurationLogger -       splunk.profiler.recording.duration : 20s
     com.splunk.opentelemetry.profiler.ConfigurationLogger -               splunk.profiler.keep-files : false
     com.splunk.opentelemetry.profiler.ConfigurationLogger -            splunk.profiler.logs-endpoint : http://<host_ip>:4318
     com.splunk.opentelemetry.profiler.ConfigurationLogger -              otel.exporter.otlp.endpoint : http://<host_ip>:4318
     com.splunk.opentelemetry.profiler.ConfigurationLogger -           splunk.profiler.memory.enabled : false
     com.splunk.opentelemetry.profiler.ConfigurationLogger -             splunk.profiler.tlab.enabled : false
     com.splunk.opentelemetry.profiler.ConfigurationLogger -        splunk.profiler.memory.event.rate : 150/s
     com.splunk.opentelemetry.profiler.ConfigurationLogger -  splunk.profiler.include.internal.stacks : false
     com.splunk.opentelemetry.profiler.ConfigurationLogger -      splunk.profiler.tracing.stacks.only : false
     com.splunk.opentelemetry.profiler.JfrActivator - Profiler is active.
     
     com.splunk.opentelemetry.profiler.EventProcessingChain - In total handled 151 events in 32ms
     io.opentelemetry.sdk.logs.export.SimpleLogRecordProcessor - Exporter failed

     Any help in understanding the root cause and resolving the export failure would be appreciated.

[bishida]

Hi,

If you set otel.exporter.otlp.endpoint then you shouldn’t have to set anything for the logs endpoint or the profiler logs endpoint because they should, by default, append /v1/logs to your otlp endpoint. It looks like you set your profiler logs endpoint but didn’t include /v1/logs which is what I think is causing your exporting error.