Hi, Quick sanity check: are you certain there's nothing being instrumented that uses an older version of .NET? The reason I ask is because the service name might be indicating .NET 6? The 1.9 instrumentation won't work with .NET6 or .NET7--you'd have to try downgrading to 1.8. "OTEL_SERVICE_NAME": "MyDotNet6WebApi" ... View more

When there are too many items to display in the legend, you should see a clickable option to "see all" and then that will open up a view below the chart with a data table containing all items.   ... View more

My understanding is that Azure API Management doesn't support OTel natively. So, I think your solution will require some creativity on the Azure side. Perhaps you could look into ways to enable traces with Application insights and then see if you can export them to OTel from there (this is just an idea--I've never tried this). You might be able to get metrics using OTel with the azuremonitor receiver. https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/azuremonitorreceiver ... View more

I think your understanding of your current scenario is all correct.  It's possible that Azure monitor has a way to create this new dimension there. So, when you export the metric through an integration like the one used by Splunk Observability Cloud, the custom DBNAME dimension is already there. That's just an idea--I don't know if Azure Monitor has a feature to do this or not, but it seems possible they might. Another possibility would be to collect this metric with an OpenTelemetry collector instead of using the Azure Cloud integration. There is a new OTel receiver being developed called azuremonitor.  https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/azuremonitorreceiver If you can collect this metric with OTel, then you can use an OTel processor to extract the short name of the database and add it to your metric using an OTel attributes processor. https://docs.splunk.com/observability/en/gdi/opentelemetry/components/attributes-processor.html ... View more

You can set a custom property such as 'the_name_of_the_database' and map it to a current dimension's name/value pair. You can do this in settings -> metric metadata. In your case, you would likely key off of "DatabaseResourceId=/SUBSCRIPTIONS/FULL_ID_OF_SUBSCRIPRTION/RESOURCEGROUPS/RESOURCE_GROUP_NAME/PROVIDERS/MICROSOFT.SQL/SERVERS/THE_NAME_OF_SQL_SERVER/DATABASES/THE_NAME_OF_DATABASE" and then set a custom property of "the_name_of_the_database=THE_NAME_OF_THE_DATABASE". To get started, search for 'DatabaseResourceId' in the metric metadata. Metric Metadata: https://docs.splunk.com/observability/en/metrics-and-metadata/metrics-finder-metadata-catalog.html You may also want to leverage the SignalFlow promote() function (depending on what you want to do with that custom property). You can filter/search without using promote() but if you want to display the_name_of_the_database on a chart, you'll likely need to use promote(). https://dev.splunk.com/observability/docs/signalflow/methods/promote_stream_method/     ... View more

Hi, The Splunk Cloud free trial instances don't have the self-service ability to set IP allow lists so you won't be able to configure Log Observer Connect with a free trial stack. If you are working with a Splunk account team, they can get you access to a proof-of-concept stack where this integration would be possible.  ... View more

Couple of possible ideas... 1. Route the alert to Splunk On-Call and use escalation policies to decide who gets alerted. 2. Choose the "webhook" option and provide a custom script that routes the alert however you want.   ... View more

Is it this one? https://dev.splunk.com/observability/reference/api/incidents/latest#endpoint-create-single-muting-rule ... View more

Hi, If you send an OpenTelemetry formatted span that contains span links, it should show up in Observability Cloud. To get this to work in the real-world, you'll probably need to manually instrument your source code. Here is an example json payload saved to a file "span.otel.json" and sent to a local OTel collector with: curl -X POST http://localhost:4318/v1/traces -H "Content-Type: application/json" -d @Span.otel.json { "resourceSpans": [ { "resource": { "attributes": [ { "key": "service.name", "value": { "stringValue": "example-service" } }, { "key": "service.instance.id", "value": { "stringValue": "instance-12345" } } ] }, "scopeSpans": [ { "scope": { "name": "example-tracer", "version": "1.0.0" }, "spans": [ { "traceId": "1e223ff1f80f1c69f8f0b81c1a2d32ad", "spanId": "6f9b3f4d1de5bf5e", "parentSpanId": "", "name": "example-operation", "kind": "SPAN_KIND_SERVER", "startTimeUnixNano": 1706100243452000000, "endTimeUnixNano": 1706100244952000000, "attributes": [ { "key": "http.method", "value": { "stringValue": "GET" } }, { "key": "http.url", "value": { "stringValue": "http://example.com/api" } } ], "links": [ { "traceId": "1e223ff1f80f1c69f8f0b81c1a2d32ae", "spanId": "9a7b3e4f1dceaa56", "attributes": [ { "key": "link-type", "value": { "stringValue": "related" } } ], "droppedAttributesCount": 0 } ] } ] } ] } ] }     ... View more

This sounds like a base use-case for the OpenTelemetry collector. When you run the OTel collector on an EC2, you’ll be streaming host metrics like cpu, memory, disk, and network directly to Splunk Observability Cloud. Since the EC2 is running in AWS, you’re also able to collect most of those same metrics through CloudWatch. The big difference is the OTel collector gives you the ability to collect high-resolution streaming metrics. This is important when correlating infrastructure metrics to application performance. The metrics coming from CloudWatch will be much lower resolution by default. But yes, this approach means that metrics are going to 2 different places: Splunk Observability Cloud and CloudWatch. The Data Management tab in Observability Cloud will give you guided instructions for installing the OTel collector. ... View more

Your application logs are stored in Splunk Cloud. Splunk Observability Cloud does not store any application logs--it uses the Log Observer Connect integration to read them from Splunk Cloud and display them. ... View more

I'm thinking the Hipster shop demo environment may be a lot of extra work to get running on MacOS/arm64.  In the following docs, they mention preferred option is to run in GKE. If you run locally, they did it with ubuntu Linux. https://github.com/GoogleCloudPlatform/microservices-demo/blob/main/docs/development-guide.md ... View more

I pulled this from one of the public examples for manual instrumentation and it worked. I think the only line I modified was the "require" line. One more note, I used .htaccess to expose env variables. require __DIR__ . '/vendor/autoload.php';   <?php declare(strict_types=1); namespace OpenTelemetry\Example; require __DIR__ . '/vendor/autoload.php'; use OpenTelemetry\Contrib\Otlp\OtlpHttpTransportFactory; use OpenTelemetry\Contrib\Otlp\SpanExporter; use OpenTelemetry\SDK\Trace\SpanProcessor\SimpleSpanProcessor; use OpenTelemetry\SDK\Trace\TracerProvider; $transport = (new OtlpHttpTransportFactory())->create('http://localhost:4318/v1/traces', 'application/x-protobuf'); $exporter = new SpanExporter($transport); echo 'Starting OTLP example'; $tracerProvider = new TracerProvider( new SimpleSpanProcessor( $exporter ) ); $tracer = $tracerProvider->getTracer('io.opentelemetry.contrib.php'); $root = $span = $tracer->spanBuilder('root')->startSpan(); $scope = $span->activate(); for ($i = 0; $i < 3; $i++) { // start a span, register some events $span = $tracer->spanBuilder('loop-' . $i)->startSpan(); $span->setAttribute('remote_ip', '1.2.3.4') ->setAttribute('country', 'USA'); $span->addEvent('found_login' . $i, [ 'id' => $i, 'username' => 'otuser' . $i, ]); $span->addEvent('generated_session', [ 'id' => md5((string) microtime(true)), ]); $span->end(); } $root->end(); $scope->detach(); echo PHP_EOL . 'OTLP example complete! '; echo PHP_EOL; $tracerProvider->shutdown(); ?>     ... View more

I suspect the OTel auto instrumentation for PHP is not very mature as of today. I experimented with it and ran in to similar challenges. When I manually instrumented PHP, it did work and I see traces that way.  ... View more

To configure log observer connect to Splunk Enterprise running on a private network, there will be additional considerations for you. You will need some help from your private networking team to allow incoming traffic from O11y Cloud. Note the IP addresses of this incoming traffic on this doc page: https://docs.splunk.com/observability/en/logs/set-up-logconnect.html#logs-set-up-logconnect A typical approach for this scenario is to use a load balancer (e.g., F5) to listen for this incoming traffic and then pass the request to the Splunk search head on your private network. Using a load balancer is nice because you can manage the ssl cert at the balancer. If you configure a true pass-through to the search head (e.g. port forwarding), then you will need to configure an ssl cert on the Splunk search head management interface which adds steps. The fact that you have an OTel collector running on your Splunk Enterprise host doesn’t affect this scenario with log observer connect. ... View more

Here is a working example of statsd receiver: After you restart the collector, it will be listening on UDP port 8125. Since this is UDP and not TCP, you can't test the port like you normally would and get a response. Send a test metric to that port and then search for it in the Metric Finder in O11y Cloud. echo "statsd.test.metric:42|c|#mykey:#myval" | nc -w 1 -u -4 localhost 8125 ... View more

I noticed one other thing that we should try. Since you're running a local instance of the OTel collector, can you unset the env variable OTEL_EXPORTER_OTLP_TRACES_HEADERS? We only want to send the token that way when you're not using a local collector. You already changed your OTEL_OTLP_EXPORTER_ENDPOINT back to http://localhost:4318, correct? Do you have your SPLUNK_ACCESS_TOKEN value set in /etc/otel/collector/splunk-otel-collector.conf? That token should have INGEST and API capabilities.  Also check that the correct secret is used in that token since you said you rotated the last one. ... View more

The messages in your /var/log/syslog appear to be a separate issue. If you want to collect logs, you can either configure your "splunk_hec" exporter to use your own Splunk HEC endpoint and token or you can disable logs for now by removing "splunk_hec" from your logs pipeline (service->pipelines->logs->exporters [remove splunk_hec]) I'm wondering if you are getting APM data into O11y Cloud but perhaps aren't generating traffic that creates spans? Can you be sure to generate some test traffic in your app that will definitely create a span? Something that calls another app or API would be ideal. You can also look for clues in the Metric Finder. Search there for the service.name you defined in your instrumentation and see if you're getting any metrics for that service.name.  ... View more

You may just want to copy/paste a small sample of anything interesting in those logs. Can you also check /var/log/syslog or /var/log/messages for potentially interesting errors coming directly from the OTel collector? ... View more

I think you have some options. You could configure a statsd receiver on your OTel collector and then send your metrics to that receiver's listening endpoint. https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/statsdreceiver If it's possible to export OTLP formatted metrics and/or logs, you could send them to the OTLP receiver's listening endpoint on your OTel collector (port 4317 for grpc or 4318 for http). For logs that you can collect from disk, you could use a Splunk universal forwarder to get your logs into Splunk Cloud or Splunk Enterprise. Or, you could use an OTel filelog receiver to collect those logs from disk and send to Splunk Cloud/Enterprise via an HEC (http event collector) endpoint. https://docs.splunk.com/observability/en/gdi/opentelemetry/components/filelog-receiver.html ... View more

OK, perhaps 4318 was correct. Can you look at logs on the host where the OTel collector is running and share any errors you find? You might see exporting errors there or some clues as to why your telemetry isn't making it to the back end. ... View more