Splunk Observability Cloud

Why am I Unable to get Kubernetes data into Splunk Observability?

Eshwar
Engager

Hi Olly experts,

We have 3 node Kubernetes Cluster node which we want to monitor so I have installed OTEL using Helm 3.0 version and it was deployed successfully but we were not able to find Kubernetes data in Olly dashboards. Please suggest how to fine tune the problem.

Note: Firewall was disabled on the server.

Regards,

Eshwar

0 Karma

bishida
Splunk Employee
Splunk Employee

Hi,

If you navigate to click "Infrastructure" on the left-hand panel, do you see a navigator tile for "K8S Nodes"?

If not, do you have more details you can share?

Do you see any logs?

Maybe you could also share the helm command you issued (just omit the token before sharing)

Is this the integration you chose? Can you share the values you used there? (please ensure the cluster name is correct for your cluster)

bishida_0-1692290611587.png

 

0 Karma

Eshwar
Engager

Hi @bishida,

Thank for your response.

I don't see K8S Nodes navigator title under Infrastructure.

You are right, I have chosen same integration for K8.

Below are the requested details:

helm install --set="splunkObservability.accessToken=xxxxxxxx,clusterName=knl-kbr-master,splunkObservability.realm=au0,gateway.enabled=false,splunkObservability.profilingEnabled=true" --generate-name splunk-otel-collector-chart/splunk-otel-collector

Eshwar_0-1692333552065.png

 

Regards,

Eshwar

0 Karma

bishida
Splunk Employee
Splunk Employee

Thanks for those details. Your helm command looks fine. Can you confirm that you're using version 3 of helm?
Also, are you able to use "kubectl logs" command to get additional clues from logs?

0 Karma

Eshwar
Engager
Hi bishida, We have used helm 3.0 to deploy the OTEL collector. I am adding logs which we could find in kubectl logs. Please suggest if you could get any clues on the issue. 2023-08-25T11:05:10.507Z info receivercreator@v0.82.0/observerhandler.go:96 starting receiver {"kind": "receiver", "name": "receiver_creator", "data_type": "metrics", "name": "smartagent/kubernetes-proxy", "endpoint": "x.x.x.x", "endpoint_id": "k8s_observer/9a7ca170-f705-4d09-add5-9e5e28bda56d"} 2023-08-25T11:05:10.511Z warn internal@v0.82.0/warning.go:40 Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks {"kind": "receiver", "name": "signalfx", "data_type": "metrics", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#..."} 2023-08-25T11:05:10.596Z info healthcheck/handler.go:129 Health Check state change {"kind": "extension", "name": "health_check", "status": "ready"} 2023-08-25T11:05:10.596Z info service/service.go:149 Everything is ready. Begin running and processing data. 2023-08-25T11:05:11.818Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:39427->x.x.x.x: write: operation not permitted", "interval": "3.195808944s"} 2023-08-25T11:05:24.360Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:49599->x.x.x.x: write: operation not permitted", "interval": "5.326381805s"} 2023-08-25T11:05:34.394Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:52489->x.x.x.x: write: operation not permitted", "interval": "6.716254549s"} 2023-08-25T11:05:44.431Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:42546->x.x.x.x: write: operation not permitted", "interval": "4.797235603s"} 2023-08-25T11:05:54.463Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:46030->x.x.x.x: write: operation not permitted", "interval": "5.719152875s"} 2023-08-25T11:06:04.651Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:45784->x.x.x.x: write: operation not permitted", "interval": "13.393955281s"} 2023-08-25T11:06:17.042Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:40316->x.x.x.x: write: operation not permitted", "interval": "9.439588945s"} 2023-08-25T11:06:31.252Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:46576->x.x.x.x: write: operation not permitted", "interval": "32.606559328s"} 2023-08-25T11:06:41.748Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:38096->x.x.x.x: write: operation not permitted", "interval": "15.181849586s"} 2023-08-25T11:06:56.427Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:51641->x.x.x.x: write: operation not permitted", "interval": "27.40037009s"} 2023-08-25T11:07:12.881Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:46738->x.x.x.x: write: operation not permitted", "interval": "42.494896094s"} 2023-08-25T11:07:23.833Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:41656->x.x.x.x: write: operation not permitted", "interval": "23.219586775s"} 2023-08-25T11:07:36.386Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:56332->x.x.x.x: write: operation not permitted", "interval": "19.881616446s"} 2023-08-25T11:07:47.055Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:35628->x.x.x.x: write: operation not permitted", "interval": "34.267612582s"} 2023-08-25T11:08:05.989Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:53651->x.x.x.x: write: operation not permitted", "interval": "41.681427935s"} 2023-08-25T11:08:18.251Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:33419->x.x.x.x: write: operation not permitted", "interval": "41.002809333s"} 2023-08-25T11:08:32.443Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:37309->x.x.x.x: write: operation not permitted", "interval": "28.231797194s"} 2023-08-25T11:08:42.814Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:47369->x.x.x.x: write: operation not permitted", "interval": "19.028467823s"} 2023-08-25T11:08:56.112Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:58940->x.x.x.x: write: operation not permitted", "interval": "20.154305214s"} 2023-08-25T11:09:06.989Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:48911->x.x.x.x: write: operation not permitted", "interval": "19.170226868s"} 2023-08-25T11:09:17.673Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:60623->x.x.x.x: write: operation not permitted", "interval": "21.717112349s"} 2023-08-25T11:09:28.412Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:53265->x.x.x.x: write: operation not permitted", "interval": "41.105192018s"} 2023-08-25T11:09:39.395Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:40975->x.x.x.x: write: operation not permitted", "interval": "22.190608963s"} 2023-08-25T11:09:53.520Z info exporterhelper/queued_retry.go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:52000->x.x.x.x: write: operation not permitted", "interval": "23.445496446s"} 2023-08-25T11:10:00.976Z error exporterhelper/queued_retry.go:165 Exporting failed. No more retries left. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": Regards, Eshwar
0 Karma

bishida
Splunk Employee
Splunk Employee

Hi,

 

This really looks like a potential firewall issue or maybe lack of DNS resolution issue.

2023-08-25T11:05:11.818Z info exporterhelper/queued_retry.
go:423 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \" https://ingest.au0.signalfx.com/v2/datapoint\": dial tcp: lookup ingest.au0.signalfx.com on x.x.x.x: write udp x.x.x.x:39427->x.x.x.x: write: operation not permitted", "interval": "3.195808944s"} 

Firewalls can sometimes block container/Kubernetes traffic. Can you please check for any possible firewall issues and make sure DNS is resolving as expected?

0 Karma
Get Updates on the Splunk Community!

Build Your First SPL2 App!

Watch the recording now!.Do you want to SPL™, too? SPL2, Splunk's next-generation data search and preparation ...

Exporting Splunk Apps

Join us on Monday, October 21 at 11 am PT | 2 pm ET!With the app export functionality, app developers and ...

[Coming Soon] Splunk Observability Cloud - Enhanced navigation with a modern look and ...

We are excited to introduce our enhanced UI that brings together AppDynamics and Splunk Observability. This is ...