In your settings -> apm and rum metricsets, you probably need to add these tags as RUM metricsets. ... View more

Hi, I think it's important to distinguish the difference between a "detector" and an "alert". You won't be able to filter detectors by host.name because a detector won't have a value for host.name. Once a detector fires an alert, then the dimensions are populated for when the matching conditions of the detector are triggered. So, you can filter alerts by host.name. Here is a screenshot of an active alert filtered by host.name.   ... View more

https://help.splunk.com/en/splunk-observability-cloud/monitor-end-user-experience/real-user-monitoring/replay-user-sessions/record-browser-sessions Is the problem with session replay? The updated instrumentation needs you to specify a value for "recorder" (just use "splunk") ... View more

I would recommend sending the alerts to the Splunk platform (Enterprise or Cloud) and performing event correlation there. This is a core use case for IT Service Intelligence. But if you wanted to just address this one example, you could do it pretty easily with a one-off search/alert in the platform.  ... View more

The time parameters should be "start" and "stop" (not "start" and "end"). Here is another command line example to try: # get cpu.utilization over past 10 minutes > STOP_MS=$(date -u +%s000) > START_MS=$(( STOP_MS - 10*60*1000 )) > REALM=us1 > TOKEN=$$$$YOUR-API-TOKEN$$$$ > curl -sS -N \ -X POST "https://stream.${REALM}.signalfx.com/v2/signalflow/execute?start=${START_MS}&stop=${STOP_MS}&resolution=10000&includeMetadata=true" \ -H "Content-Type: application/json" \ -H "X-SF-Token: ${TOKEN}" \ -d '{"programText":"data(\"cpu.utilization\").mean(over=\"1m\").publish(label=\"cpu\")"}' ... View more

Yes, there is an "execute javascript" option for a step in your test.  You should be able to write some javascript to locate the element you want to click. Specifically the querySelector() or querySelectorAll() methods should help you do that. ... View more

Check out this API reference for being able to use SignalFlow through the API: https://dev.splunk.com/observability/reference/api/signalflow/latest#endpoint-create-websocket-connection Also, if you want to work within Splunk Cloud/Enterprise, you can use the Observability Cloud Infrastructure Monitoring TA which will allow you to  use the sim command in your spl and you can use SignalFlow there to get that metric. https://splunkbase.splunk.com/app/5247 ... View more

Oops, yes, I meant to share this one. It's a native OTel receiver (as opposed to a smartagent receiver) https://help.splunk.com/en/splunk-observability-cloud/manage-data/available-data-sources/supported-integrations-in-splunk-observability-cloud/opentelemetry-receivers/rabbitmq-receiver ... View more

Hi, I think something to consider here is the intended use-case for "try now". It's designed to be a test editing validation step--not an on-demand call to run the test once to analyze results. Since you're working in a CI/CD pipeline, I'm thinking your use-case is probably the latter. I believe we have a feature on our roadmap that will better serve your use-case. You may want to ask your account rep for a roadmap call to discuss this feature with an engineer who can help determine if the upcoming feature meets your use-case. ... View more

Hi, It's probably worth noting that message is "info" level and not actually an "error". It's mostly just annoying. That directory is managed at runtime so even if you put placeholder files in there, they would get automatically deleted. As a heads-up, there is movement away from the smartagent receivers in favor of native OTel receivers as they become available. You may want to try the OTel rabbitmq receiver instead: https://help.splunk.com/en/splunk-observability-cloud/manage-data/available-data-sources/supported-integrations-in-splunk-observability-cloud/applications-messaging/rabbitmq ... View more

Hi, This is a bit of a guess—but maybe it will spark some ideas to try. I wonder if closing the computation inside of the loop is not giving the server enough time to send it’s final response. It might be worth trying introducing some delay before closing or maybe try using a “try/catch” approach when closing the computation. ... View more

OK, cool, thanks for the more info. That’s giving me more confidence that what you were originally seeing is a reflection of data roll-ups. That being said, I think the “more recent” timestamp is “correct” because the value at that timestamp represents a roll-up of the past X amount of time. For your use-case and reliably grabbing the “past minute”, I wonder if it would be a good idea to make that minute well-defined by specifying start_time and end_time instead of just “-1m” so you avoid edge cases where a datapoint might arrive late for some reason out of your control (network latency, java agent metric export, etc). So maybe the minute you query is something like from (now -2 mins) to (now -1 min). Once the data arrives at the Observability Cloud ingest endpoint, I don’t think you have to worry about any delay with ingest. The data will be recorded as it streams in even when something like a chart visualization appears to have a delay in drawing. I’d be more concerned about any potential latency from the point in time that the metric is recorded (e.g. garbage collection in the java agent) to the time it takes for the agent to export that datapoint and the time it takes for that datapoint to traverse the network to the ingest endpoint. The timestamp on the datapoint will reflect the time it was recorded even if takes extra time for that datapoint to arrive at ingest (e.g., datapoint is recorded by java agent at 16:04:01 but arrives at ingest endpoint at 16:04:45 due to some temporary network condition) ... View more

I think what is happening is that one timestamp is reflecting the end of the roll-up window and the other is reflecting the beginning. If you need them to align, you may need to subtract the value of the roll-up window to make this happen. To verify this theory, you may want to experiment with different roll-up periods and see if the difference is always equal to the roll-up. ... View more

Hi, I’m probably not understanding the question completely, so feel free to provide a specific example if you want. One thing I will point out is that when thinking about what information you can get from an inferred service--it’s limited to what you can see from the trace spans that are generated when an instrumented service calls that uninstrumented inferred service. Here is a screen shot of a service-centric view of an inferred service and what you can see about it.   ... View more

Hi, If you set otel.exporter.otlp.endpoint then you shouldn’t have to set anything for the logs endpoint or the profiler logs endpoint because they should, by default, append /v1/logs to your otlp endpoint. It looks like you set your profiler logs endpoint but didn’t include /v1/logs which is what I think is causing your exporting error.  ... View more

Since this is Windows, can you try using full paths in your command line? Example: java -javaagent:"C:\Users\user\Downloads\splunk-otel-javaagent.jar" -jar C:\Projects\my-app\target\my-app-0.0.1-SNAPSHOT.jar Also, please check if "java -version" returns the same result as "mvn -v". It's possible your call to "java" is using a different distro or version that you're not expecting. ... View more

Just to be clear, you're integrating Thousandeyes to Splunk Observability Cloud, right? If you're integrating to Splunk Enterprise or Splunk Cloud, you'll want to look at different message boards. In Splunk Observability Cloud, you can use Metric Finder to see if you're ingesting data from Thousandeyes. Look for metrics like "network.latency" or type "network." in the Metric Finder search to see if anything auto-completes. Depending on the type of Thousandeyes tests you're doing, you can also search for things like "http.server.request.availability" Thousandeyes docs might be more helpful: https://docs.thousandeyes.com/product-documentation/integration-guides/custom-built-integrations/opentelemetry/configure-opentelemetry-streams/ui ... View more

This page might help: https://docs.splunk.com/observability/en/metrics-and-metadata/relatedcontent.html#splunk-infrastructure-monitoring You may want to look at service.name, k8s.cluster.name, k8s.pod.name values to see if they're what you expect.  Other ideas to possibly consider: Are you using the Splunk Helm chart to deploy OTel to your Kubernetes cluster? Did you configure the operator in the Helm chart to instrument your Java app? Did you overwrite or change any resourcedetection processors? ... View more

Have you connected your AWS to Splunk Observability Cloud? If so, you should be able to see an ECS navigator in the Infrastructure view. https://docs.splunk.com/observability/en/gdi/get-data-in/connect/aws/get-awstoc.html   Another possible option is to check out Metric Finder and search for ecs.task.* metrics and build a chart using them and save it to a dashboard. Since you already deployed the OTel collector as a task, it should be sending in ecs.task.* metrics. ... View more

Hi, I believe you can monitor redis enterprise by including it as a custom service. Based on this documentation: https://docs.splunk.com/observability/en/gdi/integrations/cloud-azure.html I believe this to be true because this would be a "root resource type" (not a nested sub-component) and also its metrics would be exposed via Azure Monitor. ... View more

Hi, I think I’m understanding the question, but if I’m off base, just let me know. I think what I’m hearing is that you’re moving from Elastic to Splunk Observability Cloud and you’re wanting to understand how logs are exported, stored, and used in dashboards. Here is an overview of where observability data is stored and how it’s all integrated together. - Splunk Observability Cloud is where application metrics and traces are ingested and stored. - Splunk Cloud or Enterprise is where logs are ingested and stored. - Splunk Observability Cloud uses an integration called Log Observer Connect to read logs from Splunk Cloud/Enterprise and correlate them to your metrics and traces. The logs are not stored in Splunk Observability Cloud—they’re just visible through this integration. - Dashboards with logs can be created in either Splunk Observability Cloud or Splunk Cloud/Enterprise. The choice is yours and just depends on your use-case and what you want to include on the dashboards. - You may also choose to pull metrics and APM data into Splunk Cloud/Enterprise from Splunk Observability Cloud using the Splunk Infrastructure Monitoring TA. This will be helpful if you want to build your dashboards in Splunk Cloud/Enterprise and include application metrics or Real User Monitoring metrics or Synthetics test metrics. - As for getting logs into Splunk from your application, you have options: - For Kubernetes environments, I would recommend using our OpenTelemetry helm chart. You can export logs to a Splunk HEC endpoint on Splunk Cloud/Enterprise. You can also utilize OpenTelemetry pipelines to control that data any way you want. - For traditional server environments, you can simply use the Universal Forwarder to read your application logs from disk and forward them to Splunk Cloud/Enterprise. ... View more

Hi, Quick sanity check: are you certain there's nothing being instrumented that uses an older version of .NET? The reason I ask is because the service name might be indicating .NET 6? The 1.9 instrumentation won't work with .NET6 or .NET7--you'd have to try downgrading to 1.8. "OTEL_SERVICE_NAME": "MyDotNet6WebApi" ... View more

When there are too many items to display in the legend, you should see a clickable option to "see all" and then that will open up a view below the chart with a data table containing all items.   ... View more

My understanding is that Azure API Management doesn't support OTel natively. So, I think your solution will require some creativity on the Azure side. Perhaps you could look into ways to enable traces with Application insights and then see if you can export them to OTel from there (this is just an idea--I've never tried this). You might be able to get metrics using OTel with the azuremonitor receiver. https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/azuremonitorreceiver ... View more

I think your understanding of your current scenario is all correct.  It's possible that Azure monitor has a way to create this new dimension there. So, when you export the metric through an integration like the one used by Splunk Observability Cloud, the custom DBNAME dimension is already there. That's just an idea--I don't know if Azure Monitor has a feature to do this or not, but it seems possible they might. Another possibility would be to collect this metric with an OpenTelemetry collector instead of using the Azure Cloud integration. There is a new OTel receiver being developed called azuremonitor.  https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/azuremonitorreceiver If you can collect this metric with OTel, then you can use an OTel processor to extract the short name of the database and add it to your metric using an OTel attributes processor. https://docs.splunk.com/observability/en/gdi/opentelemetry/components/attributes-processor.html ... View more