Hi, Here are a couple of ideas for quick checks: 1. Did you restart the collector after changing agent_config.yaml? 2. Did you add the new apache receiver to the metrics pipeline? 3. Did you check for apache.* metrics using the metric finder? Or check for data in the apache built-in dashboard? ... View more

Hi, This error is seems to occur with older versions of helm. Can you please confirm your version of helm and see if it's possible to update to a current version? ... View more

Hi, I think there is some confusion here. The app that you linked to on SplunkBase is one that was created by a community user and not Splunk. It happens to have the word "synthetics" in the name, but that is not related to Splunk Synthetics--which is the synthetic monitoring solution provided by Splunk Observability Cloud. For help with the app you found on SplunkBase, you'll need to contact the developer directly. ... View more

It might be worth taking a look at the feature Business Workflows. It's possible they may align with your use-case. https://docs.splunk.com/observability/en/apm/workflows/workflows.html ... View more

Q1: Tokens do NOT automatically rotate. Q2: Currently, you must use the API. There is not a way to do it in the GUI. Q3: You must be an admin to rotate a token. Help Rotating a Token 1) settings -> view profile -> show user api access token (This is your personal API token. You must be an admin to continue.) 2) copy your user api token 3) go to settings -> access tokens and identify the token that should be updated (such as Default)     - Make a note of what the current value of your secret is. 4) Run a curl command from your desktop computer:     - Note: 604800 seconds is 7 days. Using a graceful value allows the old and new secret to be valid for a short term so your secret update doesn't cause any disruption.     - Replace YOUR-REALM with your organization's realm, such as us0, us1, eu0, etc.     - Replace YOUR-TOKEN-NAME with the name of the token you want to rotate (such as Default)     - Replace YOUR-USER-API-TOKEN that you noted in step 2.   curl -X  POST "https://api.YOUR-REALM.signalfx.com/v2/token/YOUR-TOKEN-NAME/rotate?graceful=604800" -H "Content-type: application/json" -H "X-SF-TOKEN: YOUR-USER-API-TOKEN"   Now your token can keep the same name, the expiration date will be refreshed to a year from now, and the secret will be new. You must copy and paste the new secret in places where you're using this token (e.g., RUM instrumentation, or OTel collector splunk-otel-collector.conf file, or Windows env variables.) Be sure to restart any collectors where you update this token secret. ... View more

Oh, wait--I see, you are using the TA. splunk-otel-collector.conf is probably not the place for this scenario. I'll have to look in to this more. ... View more

Hi, I think you can set variables like HTTP_PROXY, HTTPS_PROXY, and NO_PROXY in your splunk-otel-collector.conf file so they only apply to the collector and not the entire server. ... View more

Hi, Are you asking if you can filter spans within a trace by a time range? If so, I don't think that is possible. ... View more

Oh, since it triggered for you once but then didn't trigger again, that might be explained by the alert condition never being cleared. This could be even more likely in a test environment with little traffic. The alerts won't fire again until the previous alert condition has been cleared. There is a setting in the alert to automatically clear after X amount of time if that signal isn't reported. You might want to try that setting. Or try generating successful traffic with no errors over the period of time you're detecting on (e.g., past 15 mins). ... View more

Hi, Can you try using service.request.count as your signal (filter by sf_error:true and any other relevant filters) and see if that works?   ... View more

Hi, "error" is actually a case where you don't need to index a tag to be able to filter on it. Here is a screen shot of filtering spans where error=true. And here is an example of filtering traces that contain errors: PS - The reason it won't allow you to index "error" as a APM metricset is because "error" isn't actually a span tag so there is nothing to index.   ... View more

Awesome, glad you got it working! ... View more

Hi, Unfortunately, the error message isn't very informative. I see that you pasted your custom certificate authority cert. Can you try pasting the final cert that you signed using that CA cert/key? (this is the one that you can view/export in your web browser at https://<ip-of-splunk-enterprise-instance>:8089) Here is a quick list of other sanity checks that might help: Were you certain to specify the public IP (not a private one) of your ec2 where you see <ip-of-splunk-enterprise-instance> ? Does your AWS security policy allow incoming traffic on port 8089 from the O11y realm you're using? (e.g. us1, us0, eu0...) If you load https://<ip-of-splunk-enterprise-instance>:8089 in your browser and export the certificate, is it the same one you pasted in to the connection? (it should be) Can you login to the Splunk Enterprise instance with your service account to verify username/password is valid? Be sure the target in your LOC connection should use https and port 8089 The service account must have a role that includes capabilities "search" and "edit_tokens_own" The service account role should have a reasonable limit for searches (a multiple of 4 such as 40). ... View more

Hi, Since you’re using an IP address for the common name, did you specify IP instead of DNS in this step? ... View more

I think the db.system would have to match one of the systems listed in the supported databases list. On the backend, there is likely an "allow list" that checks if the database system is supported for Query Performance before it will show up in the UI. What is the value of your db.system when you use this clickhouse driver? ... View more

Hi, It's possible that the database you're using isn't supported for Database Query Performance. I suggest checking the supported list here: https://docs.splunk.com/observability/en/apm/db-query-perf/db-perf-reference.html#supported-dbs Also, you could check your APM MetricSets in settings->APM MetricSets and make sure that Database Query Performance is enabled and active. ... View more

Hi, These sound like APM (Application Performance Monitoring) use cases. If you can instrument the application that is making calls out to these 3 different APIs, then your application will show up as an instrumented service on the APM service map, and the 3 different APIs will show up as “inferred services”. Inferred services won’t have as much detail as an instrumented service, but you will see response times, error codes, etc that are returned when your instrumented application makes calls to them. So, yes, out-of-the-box, you will see the overview of how your application is calling these 3 other services. https://docs.splunk.com/observability/en/gdi/get-data-in/application/otel-dotnet/instrumentation/instrument-dotnet-application.html#instrument-otel-dotnet-applications ... View more

Hi, Is using status.hostIP not working for some reason?  I haven't tried it, but you might be able to just use spec.nodeName instead? ... View more

Hi, A couple of notes regarding Network Explorer. The networkExplorer data collection was deprecated in the v0.88.0 Splunk helm chart. That said, the interface from the infrastructure navigator is still available if you ingest networkExplorer data (e.g., tcp.bytes). To ingest this data, you'll probably want to consider the upstream eBpf helm chart along with the OTel collector running as a gateway. This link may help: https://docs.splunk.com/observability/en/infrastructure/network-explorer/network-explorer-setup.html#migrate-from-networkexplorer-to-ebpf-helm-chart ... View more

I'm a little suspicious of the formatting after the line: community: "public" Could you try adjusting so it's like this? community: "public" fields: - name: "some_name" oid: "x.x.x.x.x.x.x.x.x.x.x.x.x" If that doesn't help, if you could email me the agent_config.yaml, I'll take a closer look (the pdf kills the indentation). Just add "@splunk.com" to my username if you want to send it. Thanks! ... View more

Happy to take a look if you can share your agent_config.yaml file. ... View more

Hi,   Are you using the Splunk distribution of the OTel collector? You'll need it to use the smartagent receivers I think.  Here is a working example. Please note all the indentation since yaml is picky. If you want to share your agent_config.yaml, that may help.   ... View more

Hi, Here are a couple of thoughts to consider. If you want to run the OTel collector in its own container, are you using appropriate networking mode for your container? For example, port 4317 (as well as others) would need to bind to the host networking ports so that your other container running your application can refer to "localhost:4317". You may want to start the OTel collector container first and then try some simple tests on the host command line to make sure it's accessible (e.g., 'telnet localhost 4317' or 'nc -vz localhost 4317') https://lantern.splunk.com/Data_Descriptors/Docker/Setting_up_the_OpenTelemetry_Demo_in_Docker Also, I didn't see any mention of container orchestration (kubernetes). This might be a good use-case for kubernetes where you run your application container with kubernetes and the OTel collector can run in a side-car configuration. https://docs.splunk.com/observability/en/gdi/opentelemetry/collector-kubernetes/kubernetes-config.html ... View more