Hi, Couple of quick observations on your env vars. OTEL_EXPORTER_OTLP_PROTOCOL value looks suspicious. I would expect it to be "http/protobuf". OTEL_EXPORTER_OTLP_ENDPOINT value looks suspicious. port 4318 is the default port for http and 4317 is the default port for grpc. If yours is truly 4319, that means you've customized it, so maybe double check that in the case that you are running on a custom port. Otherwise, I would expect the value to be "http://localhost:4318" ... View more

Since O11y is an OpenTelemetry native solution, I believe the missing piece here is an OpenTelemetry SDK for COBOL. I think support for this is still in early discussions in the community. Once you can instrument your COBOL app with OTel, it can certainly send its traces to O11y. ... View more

I think the Precisely integration can get you some resource performance data which would be along the lines of what we think of as infrastructure metrics in O11y, but you would likely ingest those directly to Splunk Cloud/Enterprise. For O11y, we're always looking for ways to get data in via OpenTelemetry and it looks like the conversations on OpenTelemetry for mainframe are still in early stages. The same idea would apply to traces where we would want to see OpenTelemetry support for the language you're running. Conceptually, I think it's reasonable to say it's possible to instrument an application in a language like Java and point to an OTel gateway that runs outside of the mainframe. Lots of assumptions in that statement though--such as having a Java distro and version that supports auto-instrumentation and having an environment that allows that sort of data flow. ... View more

Hi, You are correct that there isn't currently a way to download RUM sessions the same way you can download an APM trace.  One possible path for you would be to collect RUM with an OTel collector and export the spans to your own system for further analysis. I haven't tried this yet personally, but with the added support for OTLP, there should be a viable way to make it work. https://help.splunk.com/en/splunk-observability-cloud/manage-data/instrument-front-end-applications/instrument-mobile-and-web-applications-for-splunk-real-user-monitoring-rum/instrument-browser-applications-for-splunk-rum/configure-the-splunk-rum-browser-agent ... View more

Hi, Couple of important points on this topic: 1) O11y Cloud doesn't ingest logs directly. Logs are ingested to Splunk Cloud/Enterprise and they are visible in O11y Cloud through Log Observer Connect integration. 2) Getting AS400 logs ingested to Splunk Cloud/Enterprise is one of those rare cases where a paid 3rd party integration is typically preferred. Check out Precisely Ironstream. ... View more

Hi, Happy to help try to break this down. In your first point, you mention that an application is hosted on the Internet and you’re unable to send traces directly to Observability Cloud. Normally, you can do exactly that. You should be able to configure your application instrumentation to send directly to Observability Cloud by setting a few env vars or javaagent flags. (e.g., SPLUNK_ACCESS_TOKEN, OTEL_EXPORTER_OTLP_TRACES_PROTOCOL, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT) Example, here is docs for how to do that with a Java app. https://help.splunk.com/en/splunk-observability-cloud/manage-data/instrument-back-end-services/instrument-back-end-applications-to-send-spans-to-splunk-apm/instrument-a-java-application/instrument-your-java-application It looks like your second point may be describing some on-prem servers or applications where you want to use an “intermediate server”. You can use a Splunk OpenTelemetry collector gateway for this. You can configure the gateway to send metrics and traces to Observability Cloud and export your logs to your Splunk Enterprise/Cloud/Heavy Forwarder. https://help.splunk.com/en/splunk-observability-cloud/manage-data/splunk-distribution-of-the-opentelemetry-collector/get-started-with-the-splunk-distribution-of-the-opentelemetry-collector ... View more

I think the most important consideration with it being categorized as "development" is that it means it's possible that collector updates could introduce breaking changes, so it's important to keep an eye on release notes and make sure your configs remain valid with new releases. The hostmetrics receivers and its associated scrapers, like "process" scraper, have pretty wide adoption and are used in production environments. ... View more

Here are a couple of thoughts that might help you choose an approach: - I'm unsure if you can limit detection to a specific timeframe only using signalflow. It might be possible--I just don't know how to do it off the top of my head, so it might be kinda tricky. - If you do want to try the muting rule approach, there is an API that could help with scaling it and automating it: https://dev.splunk.com/observability/reference/api/incidents/latest/#endpoint-create-single-muting-rule - Another idea is that you can pull metrics from Observability Cloud into Splunk Enterprise/Cloud using the Observability Cloud IM technical add-on (https://splunkbase.splunk.com/app/5247). You can use this TA to store Observability Cloud metrics in a Splunk metrics index pretty much for free (doesn't count against your ingest if you have ingest licensing). Once the metrics are in Splunk Enterprise/Cloud, the time-restricted searching and alerting would be much easier to configure. ... View more

Hi, So, quick disclaimer--there's probably a few different ways to accomplish this. But, the first idea that comes to mind for me is to take a look at muting rules. The idea here is that it might be easier to setup a detector that looks over the past 15 minutes or so and look for your custom metric filtered by job_event=START. You could detect on a static threshold where count < 1 (effectively zero). This detector would always be running, but then you strategically apply muting rules so that it only alerts you in those time windows you care about. I'm thinking you'll have to use a "custom absolute time window" but then mark it as repeating daily. https://help.splunk.com/en/splunk-observability-cloud/create-alerts-detectors-and-service-level-objectives/create-alerts-and-detectors/mute-alert-notifications ... View more

Hi, Please check out this docs page regarding the Server-Timing header: https://help.splunk.com/en/splunk-observability-cloud/monitor-end-user-experience/real-user-monitoring/introduction-to-splunk-rum One possible guess is that one of the services in that chain could be choosing to not propagate headers like Server-Timing, traceparent, etc.  ... View more

Hi, You could contact Support directly with a ticket and ask but if I had to throw out a guess, this sounds like something that would be a Professional Services task and not a support case. I might also suggest you ask your account rep if you have any on-demand service credits--if so, you might be able to use those to get some help at no additional cost. ... View more

Good callout--yes, the original attribute contains a dot (cloud.availability_zone) and then the metricset is created using only underscores. ... View more

It looks like you're on the right track and probably close to having it work. Here is an example I just tried and it seems to be working. ... View more

Hi, Creating the custom metricset looks like the correct thing to do. After you created that metricset, did you generate sufficient new traffic in that dev environment? The new metricset won't apply to old data--only data that comes in after the metricset is created. ... View more

Here are some quick sanity checks to consider: 1) You've defined the postgresql receiver in your yaml (e.g., /etc/otel/collector/agent_config.yaml) 2) You've defined the otlphttp/dbmon exporter 2a) You've inserted your realm (e.g., us0, us1, etc) in your: logs_endpoint: https://ingest.<YOUR  REALM>.signalfx.com/v3/event 3) You've added the postgresql receiver to both the metrics pipeline and the logs/dbmon pipeline 3a) You've used the same processors, in the same order, for your metrics and logs/dbmon pipelines 4) You've restarted the OTel collector Is your OTel collector running on Linux?  ... View more

Can you confirm your OTel collector is version 0.138.0 or higher? ... View more

Hi, The wildcard * in the metric name may be causing multiple MTS to be returned and all being represented as "sensu_*". Maybe try specifying a single metric name and see if that helps. If you have custom metrics in addition to other sensu metrics, you may need to create a way to filter just on those custom metrics (e.g., alter the name like sensu_mycustom_* or add a dimension that indicates it's custom) ... View more

Hi, I don't think there is a direct way to just "download all tags" but you can probably get a pretty good idea of what you have by first looking at default tags: APM tags (e.g., endpoint, operation, and HTTP method) https://help.splunk.com/en/splunk-observability-cloud/monitor-application-performance/analyze-services-with-span-tags-and-metricsets RUM tags: https://help.splunk.com/en/splunk-observability-cloud/monitor-end-user-experience/real-user-monitoring/analyze-services-with-span-tags/filter-and-troubleshoot-with-custom-tags Next, check if you have custom tags that you've created metricsets for in Settings -> APM & RUM Metricsets. Lastly, it's possible you've tagged spans using OTEL_RESOURCE_ATTRIBUTES in your instrumentation but didn't create Metricsets, so you could also check your instrumentation settings and see if you've custom tagged anything. ... View more

In your settings -> apm and rum metricsets, you probably need to add these tags as RUM metricsets. ... View more

Hi, I think it's important to distinguish the difference between a "detector" and an "alert". You won't be able to filter detectors by host.name because a detector won't have a value for host.name. Once a detector fires an alert, then the dimensions are populated for when the matching conditions of the detector are triggered. So, you can filter alerts by host.name. Here is a screenshot of an active alert filtered by host.name.   ... View more

https://help.splunk.com/en/splunk-observability-cloud/monitor-end-user-experience/real-user-monitoring/replay-user-sessions/record-browser-sessions Is the problem with session replay? The updated instrumentation needs you to specify a value for "recorder" (just use "splunk") ... View more

I would recommend sending the alerts to the Splunk platform (Enterprise or Cloud) and performing event correlation there. This is a core use case for IT Service Intelligence. But if you wanted to just address this one example, you could do it pretty easily with a one-off search/alert in the platform.  ... View more

The time parameters should be "start" and "stop" (not "start" and "end"). Here is another command line example to try: # get cpu.utilization over past 10 minutes > STOP_MS=$(date -u +%s000) > START_MS=$(( STOP_MS - 10*60*1000 )) > REALM=us1 > TOKEN=$$$$YOUR-API-TOKEN$$$$ > curl -sS -N \ -X POST "https://stream.${REALM}.signalfx.com/v2/signalflow/execute?start=${START_MS}&stop=${STOP_MS}&resolution=10000&includeMetadata=true" \ -H "Content-Type: application/json" \ -H "X-SF-Token: ${TOKEN}" \ -d '{"programText":"data(\"cpu.utilization\").mean(over=\"1m\").publish(label=\"cpu\")"}' ... View more