Nothing shows up in always on profilin in Splunk observability. using 14 day free trial

[AjinkyaNerkar]

Hey Can someone help me with getting the profiling metrics like cpu and ram used by the app to show up in the splunk observation portal , I can get the app metrics so i have used a simple curl java app which curls google every second and this shows up in apm metrics I have done all the configs to have the profiling enabled as per all teh docs in splunk but nothing shows up in the profiling section . is it because i am using the free trial ? 

I am trying this on simple ec2 instance instrumenting the app using java jar command and I have been exporting the necessary vars and have added the required java options while instrumenting the app using splunk-otel-agent-collector.jar but nothing shows up please help.

[bishida]

Hi,

AlwaysOn Profiling should work in the trial and with Java. Can you please share what version of Java and what distribution you're using? It would also be helpful to know what you have configured as your Java arguments for "-javaagent" and any environment variables you may have set. 

[AjinkyaNerkar]

Hey Bishida, 

Thanks for your prompt reply! 

Java version:  jdk11 on ubuntu sandbox, "1.8.0_392" on local, OpenJDK version "17.0.10" 2024-01-16 on a k8s pod 
We have used ubuntu ec2, pods with java, WSL on Linux but no luck on profiling on any of those. 
But none of them are able to show up profiling on the Splunk observability platform. 

Then we are instrumenting the curl app using the given command 

java -javaagent:splunk-otel-javaagent.jar -Dsplunk.profiler.enabled=true -Dsplunk.profiler.memory.enabled=true -Dotel.service.name=ec2 -Dotel.resource.attributes=deployment.environment=lab,service.version=1.0 -Dotel.exporter.otlp.endpoint=http://localhost:4317 -Dsplunk.metrics.endpoint=http://localhost:9943 -jar target/curlapp-1.1-SNAPSHOT.jar

and setting up the following Gvars.

SPLUNK_OTEL_AGENT=localhost
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
OTEL_SERVICE_NAME="java_application"
OTEL_RESOURCE_ATTRIBUTES="deployment.environment=lab,service.version=1.0"

So, we are using a curl app https://github.com/astro7982/curlappstatic this was present in Splunk's one of the tutorials. 

Then we are installing the collector, I have tried various ways one is adding integration and copying the commands and it starts using the systemctl in an ubuntu image.  

One thing is that after instrumenting the app we get the app logs with some otel  populated 
metrics like "profiling JFR detected at "./jfr" file name.
Also, apart from putting on the service name and env on exporting vars and passing java options logs says that no service name is set "and asks us to export it again, we have cross checked the service name is set properly and on APM we can the service name too. 

We have also tried this with simple hello world java app and with petclinic but no luck. 

Hope this information is useful! 

[bishida]

Hi,

Here is something you could check. I see in your command line, you have 

-javaagent:splunk-otel-javaagent.jar

 
That probably works as long as your current working directory is where that jar file lives. You could try specifying the full path there instead to be sure. Then, you'll want to make sure that the user that is running your Java app has read/write permissions to that directory where the splunk-otel-javaagent.jar lives because that will be the default location of the profiler reading and writing the jfr files.

I assume you've already confirmed that the OTel collector is running and sending other types of data to the Observability Cloud (such as host metrics).