Hello Splunk Community, I have a use case where we need to send metrics directly to Splunk instead of AWS CloudWatch, while still sending CPU and memory metrics to CloudWatch for auto-scaling purposes. Datadog offers solutions, such as their AgentCheck package (https://docs.datadoghq.com/developers/custom_checks/write_agent_check/), and their repository (https://github.com/DataDog/integrations-core) provides several integrations for similar use cases. Is there an equivalent solution or approach available in Splunk for achieving this functionality? Looking forward to your suggestions and guidance! Thanks! ... View more

I am using StatsD to send metrics to a receiver, but I am encountering an issue where timing metrics (|ms) are not being captured, even though counter metrics (|c) work fine on Splunk Observability Cloud.   Example of Working Metric: The following command works and is processed correctly by the StatsD receiver:   echo "test_Latency:42|c|#key:val" | nc -u -w1 localhost 8127   Example of Non-Working Metric: However, this command does not result in any output or processing:   echo "test_Latency:0.082231|ms" | nc -u -w1 localhost 8127   Current StatsD Configuration: Here is the configuration I am using for the receiver by following the doc: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/statsdreceiver   receivers: statsd: endpoint: "localhost:8127" aggregation_interval: 30s enable_metric_type: true is_monotonic_counter: false timer_histogram_mapping: - statsd_type: "histogram" observer_type: "gauge" - statsd_type: "timing" observer_type: "histogram" histogram: max_size: 100 - statsd_type: "distribution" observer_type: "summary" summary: percentiles: [0, 10, 50, 90, 95, 100]    Why are timing metrics (|ms) not being captured while counters (|c) are working, can you please help to check on it as the statsdreceiver github document says it supports "timer" related metrics https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/statsdreceiver/README.md#timer Any help or suggestions would be greatly appreciated. Thank You. ... View more

Hello, I am following document: https://docs.splunk.com/Documentation/Splunk/9.4.0/Security/ConfigureandinstallcertificatesforLogObserver?ref=hk to configure and install certificates in Splunk Enterprise for Splunk Log Observer Connect but getting some error mentioned below. I have generated myFinalCert.pem as per the document mentioned above, below are the server.conf and web.conf configuration. # cat ../etc/system/local/server.conf [general] serverName = ip-xxxx.us-west-2.compute.internal pass4SymmKey = $7$IHXMpPIvtTGnxEusRYk62AjBIizAQosZq0YXtUg== [sslConfig] serverCert = /opt/splunk/etc/auth/sloccerts/myFinalCert.pem requireClientCert = false sslPassword = $7$vboieDG2v4YFg8FbYxW8jDji6woyDylOKWLe8Ow== [lmpool:auto_generated_pool_download-trial] description = auto_generated_pool_download-trial peers = * quota = MAX stack_id = download-trial [lmpool:auto_generated_pool_forwarder] description = auto_generated_pool_forwarder peers = * quota = MAX stack_id = forwarder [lmpool:auto_generated_pool_free] description = auto_generated_pool_free peers = * quota = MAX stack_id = free # cat ../etc/system/local/web.conf [expose:tlPackage-scimGroup] methods = GET pattern = /identity/provisioning/v1/scim/v2/Groups/* [expose:tlPackage-scimGroups] methods = GET pattern = /identity/provisioning/v1/scim/v2/Groups [expose:tlPackage-scimUser] methods = GET,PUT,PATCH,DELETE pattern = /identity/provisioning/v1/scim/v2/Users/* [expose:tlPackage-scimUsers] methods = GET pattern = /identity/provisioning/v1/scim/v2/Users [settings] enableSplunkWebSSL = true serverCert = /opt/splunk/etc/auth/sloccerts/myFinalCert.pem # After making changes to server.conf, I am able to restart the splunkd service but after making changes to the web.conf, restarting the splunkd service gets stuck, below are logs related to it: # ./splunk restart splunkd is not running. [FAILED] Splunk> The IT Search Engine. Checking prerequisites... Checking http port [8000]: open Checking mgmt port [8089]: open Checking appserver port [127.0.0.1:8065]: open Checking kvstore port [8191]: open Checking configuration... Done. Checking critical directories... Done Checking indexes... Validated: _audit _configtracker _dsappevent _dsclient _dsphonehome _internal _introspection _metrics _metrics_rollup _telemetry _thefishbucket history main sim_metrics statsd_udp_8125_5_dec summary Done Checking filesystem compatibility... Done Checking conf files for problems... Done Checking default conf files for edits... Validating installed files against hashes from '/opt/splunk/splunk-9.3.2-d8bb32809498-linux-2.6-x86_64-manifest' All installed files intact. Done All preliminary checks passed. Starting splunk server daemon (splunkd)... PYTHONHTTPSVERIFY is set to 0 in splunk-launch.conf disabling certificate validation for the httplib and urllib libraries shipped with the embedded Python interpreter; must be set to "1" for increased security Done [ OK ] Waiting for web server at https://127.0.0.1:8000 to be available...............................WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details. Please let me know if I am missing some thing. Thanks ... View more

Hello, I am configuring statsd to send custom metric from AWS EC2 instance on which splunk-otel-collector.service is running to Splunk Observability Cloud to monitor this custom metrics. I have followed the steps mentioned in the https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/statsdreceiver to setup statsd as receiver. receivers: statsd: endpoint: "localhost:8125" # default aggregation_interval: 60s # default enable_metric_type: false # default is_monotonic_counter: false # default timer_histogram_mapping: - statsd_type: "histogram" observer_type: "histogram" histogram: max_size: 50 - statsd_type: "distribution" observer_type: "histogram" histogram: max_size: 50 - statsd_type: "timing" observer_type: "summary" I have a problem in setting service for this statsd as receiver, as per github doc below configuration is written for the exporters, but I am not sure how this will work. exporters: file: path: ./test.json service: pipelines: metrics: receivers: [statsd] exporters: [file]  I also tried setting exporters in service section "receivers: [hostmetrics, otlp, signalfx, statsd]" and "exporters: [signalfx]" in the agent_config.yaml file as mentioned below, when I restart the "systemctl restart splunk-otel-collector.service", splunk otel collector agent stop sending any metric to the  Splunk Observability Cloud and when I remove statsd (receivers: [hostmetrics, otlp, signalfx]) then splunk otel collector agent starts sending any metric to the  Splunk Observability Cloud. # pwd /etc/otel/collector # # ls agent_config.yaml config.d fluentd gateway_config.yaml splunk-otel-collector.conf splunk-otel-collector.conf.example splunk-support-bundle.sh # service: extensions: [health_check, http_forwarder, zpages, smartagent] pipelines: traces: receivers: [jaeger, otlp, zipkin] processors: - memory_limiter - batch - resourcedetection #- resource/add_environment exporters: [otlphttp, signalfx] # Use instead when sending to gateway #exporters: [otlp/gateway, signalfx] metrics: receivers: [hostmetrics, otlp, signalfx, statsd] processors: [memory_limiter, batch, resourcedetection] exporters: [signalfx] # Use instead when sending to gateway #exporters: [otlp/gateway] What should be correct/supported exporter for the statsd as receiver? Thanks ... View more

Hello Team, I have successfully set up Splunk Observability Cloud to monitor Amazon Web Services through Amazon CloudWatch and can now observe all AWS services via IAM role. Additionally, I have a gRPC application running on an AWS EC2 instance, which generates custom metrics using a StatsD server via Golang. I would like to send these custom metrics to Splunk Observability Cloud to monitor the health of the gRPC application, along with the logs it generates. On my AWS Linux machine, I can see that the host monitoring agent is installed and the splunk-otel-collector service is running. Could you please advise on the method to send the custom metrics and logs generated by the StatsD server from the Golang gRPC application to Splunk Observability Cloud for monitoring? Thank you. ... View more

Hello, Getting Action forbidden error when going to "https://<hostname>/en-US/app/search/analytics_workspace" on Splunk Cloud. Please note: I have logged with sc_admin role. Thanks ... View more

Hello Community, I am trying to create a connection so that I can sent metric running on 8125 port UDP on Splunk Enterprise (running locally) to Spunk Cloud (running prd-p-7mh2z.splunkcloud.com) but I am getting below error. As I need to send UDP data running on port 8125, I am using heavy forwarder instead of universal forwarder and I have configured heavy forwarder pointing to "prd-p-7mh2z.splunkcloud.com:9997" Getting error on the dashboard ``` The TCP output processor has paused the data flow. Forwarding to host_dest=prd-p-7mh2z.splunkcloud.com inside output group default-autolb-group from host_src=rahusri2s-MacBook-Pro.local has been blocked for blocked_seconds=10. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data. ```   cat /Applications/splunk/etc/system/local/outputs.conf Password: [tcpout] defaultGroup = default-autolb-group indexAndForward = 1 [tcpout:default-autolb-group] server = prd-p-7mh2z.splunkcloud.com:9997 [tcpout-server://prd-p-7mh2z.splunkcloud.com:9997] # cat /Applications/splunk/etc/apps/search/local/inputs.conf [splunktcp://9997] connection_host = ip [udp://8125] connection_host = dns host = rahusri2s-MacBook-Pro.local index = 4_dec_8125_udp sourcetype = statsd   Thanks in advance. #splunk  ... View more