Is there any other option which  can be monitor tomcat?  Please suggest! Regards, Eshwar ... View more

Hi, I had followed below documentation. https://docs.splunk.com/observability/en/gdi/monitors-hosts/apache-tomcat.html There was mentioned Kubernetes and Linux in the installation steps. Not able to monitor tomcat via OTEL collector. I am getting below error in event logs.   Regards, Eshwar ... View more

Hi @richgalloway , I got it but thing here is I want to get those events which are retrieving via search through REST API because we are integrating Splunk with other tool to forward all the events. So, we are looking for an API which provides all events. Regards, Eshwar ... View more

Hi @scelikok , That's working great. Thank you for saving my time. Regards, Eshwar   ... View more

Hi @scelikok , I tried with output_mode=json but not able to get JSON response as my REST end point is for fired alerts as below. https://localhost:8089/services/alerts/fired_alerts ... View more

Hi @richgalloway , We had tried giving the output_mode parameter for Rest point but still we can see xml  response. ... View more

Hi bishida, We have used helm 3.0 to deploy the OTEL collector. I am adding logs which we could find in kubectl logs. Please suggest if you could get any clues on the issue. 2023-08-25T11:05:10.507Z info receivercreator@v0.82.0/observerhandler.go:96 starting receiver {"kind": "receiver", "name": "receiver_creator", "data_type": "metrics", "name": "smartagent/kubernetes-proxy", "endpoint": "x.x.x.x", "endpoint_id": "k8s_observer/9a7ca170-f705-4d09-add5-9e5e28bda56d"} 2023-08-25T11:05:10.511Z warn internal@v0.82.0/warning.go:40 Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks {"kind": "receiver", "name": "signalfx", "data_type": "metrics", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"} 2023-08-25T11:05:10.596Z info healthcheck/handler.go:129 Health Check state change {"kind": "extension", "name": "health_check", "status": "ready"} 2023-08-25T11:05:10.596Z info service/service.go:149 Everything is ready. Begin running and processing data. 2023-08-25T11:05:11.818Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:39427->x.x.x.x: write: operation not permitted", "interval": "3.195808944s"} 2023-08-25T11:05:24.360Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:49599->x.x.x.x: write: operation not permitted", "interval": "5.326381805s"} 2023-08-25T11:05:34.394Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:52489->x.x.x.x: write: operation not permitted", "interval": "6.716254549s"} 2023-08-25T11:05:44.431Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:42546->x.x.x.x: write: operation not permitted", "interval": "4.797235603s"} 2023-08-25T11:05:54.463Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:46030->x.x.x.x: write: operation not permitted", "interval": "5.719152875s"} 2023-08-25T11:06:04.651Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:45784->x.x.x.x: write: operation not permitted", "interval": "13.393955281s"} 2023-08-25T11:06:17.042Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:40316->x.x.x.x: write: operation not permitted", "interval": "9.439588945s"} 2023-08-25T11:06:31.252Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:46576->x.x.x.x: write: operation not permitted", "interval": "32.606559328s"} 2023-08-25T11:06:41.748Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:38096->x.x.x.x: write: operation not permitted", "interval": "15.181849586s"} 2023-08-25T11:06:56.427Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:51641->x.x.x.x: write: operation not permitted", "interval": "27.40037009s"} 2023-08-25T11:07:12.881Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:46738->x.x.x.x: write: operation not permitted", "interval": "42.494896094s"} 2023-08-25T11:07:23.833Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:41656->x.x.x.x: write: operation not permitted", "interval": "23.219586775s"} 2023-08-25T11:07:36.386Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:56332->x.x.x.x: write: operation not permitted", "interval": "19.881616446s"} 2023-08-25T11:07:47.055Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:35628->x.x.x.x: write: operation not permitted", "interval": "34.267612582s"} 2023-08-25T11:08:05.989Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:53651->x.x.x.x: write: operation not permitted", "interval": "41.681427935s"} 2023-08-25T11:08:18.251Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:33419->x.x.x.x: write: operation not permitted", "interval": "41.002809333s"} 2023-08-25T11:08:32.443Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:37309->x.x.x.x: write: operation not permitted", "interval": "28.231797194s"} 2023-08-25T11:08:42.814Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:47369->x.x.x.x: write: operation not permitted", "interval": "19.028467823s"} 2023-08-25T11:08:56.112Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:58940->x.x.x.x: write: operation not permitted", "interval": "20.154305214s"} 2023-08-25T11:09:06.989Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:48911->x.x.x.x: write: operation not permitted", "interval": "19.170226868s"} 2023-08-25T11:09:17.673Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:60623->x.x.x.x: write: operation not permitted", "interval": "21.717112349s"} 2023-08-25T11:09:28.412Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:53265->x.x.x.x: write: operation not permitted", "interval": "41.105192018s"} 2023-08-25T11:09:39.395Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:40975->x.x.x.x: write: operation not permitted", "interval": "22.190608963s"} 2023-08-25T11:09:53.520Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:52000->x.x.x.x: write: operation not permitted", "interval": "23.445496446s"} 2023-08-25T11:10:00.976Z error exporterhelper/queued_retry.go:165 Exporting failed. No more retries left. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": Regards, Eshwar ... View more

Hello experts, Any Advice on above query? Regards, Eshwar ... View more

Hi @bishida, Thank for your response. I don't see K8S Nodes navigator title under Infrastructure. You are right, I have chosen same integration for K8. Below are the requested details: helm install --set="splunkObservability.accessToken=xxxxxxxx,clusterName=knl-kbr-master,splunkObservability.realm=au0,gateway.enabled=false,splunkObservability.profilingEnabled=true" --generate-name splunk-otel-collector-chart/splunk-otel-collector   Regards, Eshwar ... View more

Hi @richgalloway ,  We are forwarding the data to Cloud instance from HWF but we don't see any data on Cloud instance. Can you suggest how to remove the blocking queues in HWF as my understand disable is the option right? Regards, Eshwar ... View more

Hi @isoutamo , It is a classic experience.  ... View more

Yes, User has sc_admin role even though can't see the option. Regards, Eshwar ... View more