Hi @sainag_splunk  I have reconfigured HEC and I am able to send data to HEC indexer via Post man. Since, I had configured OTEL collector according to HEC but I am not able to see data from OTEL collector. Can you please suggest where went wrong. Thank you in advance. Regards, Eshwar ... View more

Hi @sainag_splunk , I am trying to open the end point on browser but getting below error. Regards, Eshwar ... View more

Hi @sainag_splunk , Yes, I had configured with token, index. Below is my configuration in HEC and OTEL exporter.   Please suggest where went wrong? Regards, Eshwar ... View more

Hi @sainag_splunk , Thank you for your response. Just for your info I had installed HEX on on-prem not on Kubernetes. I think that command you have shared is for Kubernetes environment. My goal is to achieve sending log data through Otel collector to HEC end point. ... View more

Is there any other option which  can be monitor tomcat?  Please suggest! Regards, Eshwar ... View more

Hi, I had followed below documentation. https://docs.splunk.com/observability/en/gdi/monitors-hosts/apache-tomcat.html There was mentioned Kubernetes and Linux in the installation steps. Not able to monitor tomcat via OTEL collector. I am getting below error in event logs.   Regards, Eshwar ... View more

Hi @richgalloway , I got it but thing here is I want to get those events which are retrieving via search through REST API because we are integrating Splunk with other tool to forward all the events. So, we are looking for an API which provides all events. Regards, Eshwar ... View more

Hi @scelikok , That's working great. Thank you for saving my time. Regards, Eshwar   ... View more

Hi @scelikok , I tried with output_mode=json but not able to get JSON response as my REST end point is for fired alerts as below. https://localhost:8089/services/alerts/fired_alerts ... View more

Hi @richgalloway , We had tried giving the output_mode parameter for Rest point but still we can see xml  response. ... View more

Hi bishida, We have used helm 3.0 to deploy the OTEL collector. I am adding logs which we could find in kubectl logs. Please suggest if you could get any clues on the issue. 2023-08-25T11:05:10.507Z info receivercreator@v0.82.0/observerhandler.go:96 starting receiver {"kind": "receiver", "name": "receiver_creator", "data_type": "metrics", "name": "smartagent/kubernetes-proxy", "endpoint": "x.x.x.x", "endpoint_id": "k8s_observer/9a7ca170-f705-4d09-add5-9e5e28bda56d"} 2023-08-25T11:05:10.511Z warn internal@v0.82.0/warning.go:40 Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks {"kind": "receiver", "name": "signalfx", "data_type": "metrics", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"} 2023-08-25T11:05:10.596Z info healthcheck/handler.go:129 Health Check state change {"kind": "extension", "name": "health_check", "status": "ready"} 2023-08-25T11:05:10.596Z info service/service.go:149 Everything is ready. Begin running and processing data. 2023-08-25T11:05:11.818Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:39427->x.x.x.x: write: operation not permitted", "interval": "3.195808944s"} 2023-08-25T11:05:24.360Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:49599->x.x.x.x: write: operation not permitted", "interval": "5.326381805s"} 2023-08-25T11:05:34.394Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:52489->x.x.x.x: write: operation not permitted", "interval": "6.716254549s"} 2023-08-25T11:05:44.431Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:42546->x.x.x.x: write: operation not permitted", "interval": "4.797235603s"} 2023-08-25T11:05:54.463Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:46030->x.x.x.x: write: operation not permitted", "interval": "5.719152875s"} 2023-08-25T11:06:04.651Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:45784->x.x.x.x: write: operation not permitted", "interval": "13.393955281s"} 2023-08-25T11:06:17.042Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:40316->x.x.x.x: write: operation not permitted", "interval": "9.439588945s"} 2023-08-25T11:06:31.252Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:46576->x.x.x.x: write: operation not permitted", "interval": "32.606559328s"} 2023-08-25T11:06:41.748Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:38096->x.x.x.x: write: operation not permitted", "interval": "15.181849586s"} 2023-08-25T11:06:56.427Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:51641->x.x.x.x: write: operation not permitted", "interval": "27.40037009s"} 2023-08-25T11:07:12.881Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:46738->x.x.x.x: write: operation not permitted", "interval": "42.494896094s"} 2023-08-25T11:07:23.833Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:41656->x.x.x.x: write: operation not permitted", "interval": "23.219586775s"} 2023-08-25T11:07:36.386Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:56332->x.x.x.x: write: operation not permitted", "interval": "19.881616446s"} 2023-08-25T11:07:47.055Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:35628->x.x.x.x: write: operation not permitted", "interval": "34.267612582s"} 2023-08-25T11:08:05.989Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:53651->x.x.x.x: write: operation not permitted", "interval": "41.681427935s"} 2023-08-25T11:08:18.251Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:33419->x.x.x.x: write: operation not permitted", "interval": "41.002809333s"} 2023-08-25T11:08:32.443Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:37309->x.x.x.x: write: operation not permitted", "interval": "28.231797194s"} 2023-08-25T11:08:42.814Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:47369->x.x.x.x: write: operation not permitted", "interval": "19.028467823s"} 2023-08-25T11:08:56.112Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:58940->x.x.x.x: write: operation not permitted", "interval": "20.154305214s"} 2023-08-25T11:09:06.989Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:48911->x.x.x.x: write: operation not permitted", "interval": "19.170226868s"} 2023-08-25T11:09:17.673Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:60623->x.x.x.x: write: operation not permitted", "interval": "21.717112349s"} 2023-08-25T11:09:28.412Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:53265->x.x.x.x: write: operation not permitted", "interval": "41.105192018s"} 2023-08-25T11:09:39.395Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:40975->x.x.x.x: write: operation not permitted", "interval": "22.190608963s"} 2023-08-25T11:09:53.520Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:52000->x.x.x.x: write: operation not permitted", "interval": "23.445496446s"} 2023-08-25T11:10:00.976Z error exporterhelper/queued_retry.go:165 Exporting failed. No more retries left. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": Regards, Eshwar ... View more

Hello experts, Any Advice on above query? Regards, Eshwar ... View more

Hi @bishida, Thank for your response. I don't see K8S Nodes navigator title under Infrastructure. You are right, I have chosen same integration for K8. Below are the requested details: helm install --set="splunkObservability.accessToken=xxxxxxxx,clusterName=knl-kbr-master,splunkObservability.realm=au0,gateway.enabled=false,splunkObservability.profilingEnabled=true" --generate-name splunk-otel-collector-chart/splunk-otel-collector   Regards, Eshwar ... View more

Hi @richgalloway ,  We are forwarding the data to Cloud instance from HWF but we don't see any data on Cloud instance. Can you suggest how to remove the blocking queues in HWF as my understand disable is the option right? Regards, Eshwar ... View more