Why am I Unable to get Kubernetes data into Splunk Observability?

[Eshwar]

Hi Olly experts,

We have 3 node Kubernetes Cluster node which we want to monitor so I have installed OTEL using Helm 3.0 version and it was deployed successfully but we were not able to find Kubernetes data in Olly dashboards. Please suggest how to fine tune the problem.

Note: Firewall was disabled on the server.

Regards,

Eshwar

[bishida]

Hi,

If you navigate to click "Infrastructure" on the left-hand panel, do you see a navigator tile for "K8S Nodes"?

If not, do you have more details you can share?

Do you see any logs?

Maybe you could also share the helm command you issued (just omit the token before sharing)

Is this the integration you chose? Can you share the values you used there? (please ensure the cluster name is correct for your cluster)

[Eshwar]

Hi @bishida,

Thank for your response.

I don't see K8S Nodes navigator title under Infrastructure.

You are right, I have chosen same integration for K8.

Below are the requested details:

helm install --set="splunkObservability.accessToken=xxxxxxxx,clusterName=knl-kbr-master,splunkObservability.realm=au0,gateway.enabled=false,splunkObservability.profilingEnabled=true" --generate-name splunk-otel-collector-chart/splunk-otel-collector

Regards,

Eshwar

[bishida]

Thanks for those details. Your helm command looks fine. Can you confirm that you're using version 3 of helm?
Also, are you able to use "kubectl logs" command to get additional clues from logs?

[Eshwar]

Hi bishida, We have used helm 3.0 to deploy the OTEL collector. I am adding logs which we could find in kubectl logs. Please suggest if you could get any clues on the issue. 2023-08-25T11:05:10.507Z info receivercreator@v0.82.0/observerhandler.go:96 starting receiver {"kind": "receiver", "name": "receiver_creator", "data_type": "metrics", "name": "smartagent/kubernetes-proxy", "endpoint": "x.x.x.x", "endpoint_id": "k8s_observer/9a7ca170-f705-4d09-add5-9e5e28bda56d"} 2023-08-25T11:05:10.511Z warn internal@v0.82.0/warning.go:40 Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks {"kind": "receiver", "name": "signalfx", "data_type": "metrics", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#..."} 2023-08-25T11:05:10.596Z info healthcheck/handler.go:129 Health Check state change {"kind": "extension", "name": "health_check", "status": "ready"} 2023-08-25T11:05:10.596Z info service/service.go:149 Everything is ready. Begin running and processing data. 2023-08-25T11:05:11.818Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:39427->x.x.x.x: write: operation not permitted", "interval": "3.195808944s"} 2023-08-25T11:05:24.360Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:49599->x.x.x.x: write: operation not permitted", "interval": "5.326381805s"} 2023-08-25T11:05:34.394Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:52489->x.x.x.x: write: operation not permitted", "interval": "6.716254549s"} 2023-08-25T11:05:44.431Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:42546->x.x.x.x: write: operation not permitted", "interval": "4.797235603s"} 2023-08-25T11:05:54.463Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:46030->x.x.x.x: write: operation not permitted", "interval": "5.719152875s"} 2023-08-25T11:06:04.651Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:45784->x.x.x.x: write: operation not permitted", "interval": "13.393955281s"} 2023-08-25T11:06:17.042Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:40316->x.x.x.x: write: operation not permitted", "interval": "9.439588945s"} 2023-08-25T11:06:31.252Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:46576->x.x.x.x: write: operation not permitted", "interval": "32.606559328s"} 2023-08-25T11:06:41.748Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:38096->x.x.x.x: write: operation not permitted", "interval": "15.181849586s"} 2023-08-25T11:06:56.427Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:51641->x.x.x.x: write: operation not permitted", "interval": "27.40037009s"} 2023-08-25T11:07:12.881Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:46738->x.x.x.x: write: operation not permitted", "interval": "42.494896094s"} 2023-08-25T11:07:23.833Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:41656->x.x.x.x: write: operation not permitted", "interval": "23.219586775s"} 2023-08-25T11:07:36.386Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:56332->x.x.x.x: write: operation not permitted", "interval": "19.881616446s"} 2023-08-25T11:07:47.055Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:35628->x.x.x.x: write: operation not permitted", "interval": "34.267612582s"} 2023-08-25T11:08:05.989Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:53651->x.x.x.x: write: operation not permitted", "interval": "41.681427935s"} 2023-08-25T11:08:18.251Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:33419->x.x.x.x: write: operation not permitted", "interval": "41.002809333s"} 2023-08-25T11:08:32.443Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:37309->x.x.x.x: write: operation not permitted", "interval": "28.231797194s"} 2023-08-25T11:08:42.814Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:47369->x.x.x.x: write: operation not permitted", "interval": "19.028467823s"} 2023-08-25T11:08:56.112Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:58940->x.x.x.x: write: operation not permitted", "interval": "20.154305214s"} 2023-08-25T11:09:06.989Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:48911->x.x.x.x: write: operation not permitted", "interval": "19.170226868s"} 2023-08-25T11:09:17.673Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:60623->x.x.x.x: write: operation not permitted", "interval": "21.717112349s"} 2023-08-25T11:09:28.412Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:53265->x.x.x.x: write: operation not permitted", "interval": "41.105192018s"} 2023-08-25T11:09:39.395Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:40975->x.x.x.x: write: operation not permitted", "interval": "22.190608963s"} 2023-08-25T11:09:53.520Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:52000->x.x.x.x: write: operation not permitted", "interval": "23.445496446s"} 2023-08-25T11:10:00.976Z error exporterhelper/queued_retry.go:165 Exporting failed. No more retries left. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": Regards, Eshwar

[bishida]

Hi,

This really looks like a potential firewall issue or maybe lack of DNS resolution issue.

2023-08-25T11:05:11.818Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \" https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:39427->x.x.x.x: write: operation not permitted", "interval": "3.195808944s"} 

Firewalls can sometimes block container/Kubernetes traffic. Can you please check for any possible firewall issues and make sure DNS is resolving as expected?