Kubernetes Observability - 12/10/24

ArifV — Thu, 12 Dec 2024 19:32:35 GMT

Register here.This thread is for the Community Office Hours session on Kubernetes Observability on Tue, December 10, 2024 at 1pm PT / 4pm ET.

What can I ask in this AMA?

How do I use and customize Kubernetes navigators?
What are best practices for optimizing Kubernetes alerts and troubleshooting workflows?
Is there a way to view Kubernetes logs correlated with metrics?
How do I review Pod status?
How do I monitor Kubernetes resource limits?
Anything else you’d like to learn!

Please submit your questions at registration. You can also head to the#office-hours user Slack channel to ask questions (request access here).

Pre-submitted questions will be prioritized. After that, we will open the floor up to live Q&A with meeting participants.

Look forward to connecting!

Re: Kubernetes Observability - 12/10/24

ArifV — Thu, 12 Dec 2024 19:32:09 GMT

Here are some of the questions covered in the session:

Q1: How do you get logs from a Kubernetes Cluster?

Solution:

Set Log Collection in the wizard:

See documentation - covers cases like host logs, multi-line logs, using pod annotations, sending events, etc.
There are also many more tips about this topic from slides 8-14 in this deck

Documentation:

https://docs.splunk.com/observability/en/gdi/opentelemetry/collector-kubernetes/kubernetes-config-logs.html

Q2: How can I optimize troubleshooting for K8s alerts?

Solution:

Utilize the Navigator links embedded in alerts
Autodetector alerts
Utilize built-in metrics and dashboards
Identify problematic nodes, pods, and containers using the hierarchy map in the Kubernetes Navigator
Deploy the Splunk Distribution of the Otel Collector to your cluster for correlation of metrics, traces and logs (related content)
Enrich telemetry data with custom metrics or adding relevant metadata to enhance troubleshooting

Documentation:

Q3: One of my containers produces events as json. Any pointers on how to teach the OTel Collector to read it in as json, I am getting multiple events strung together as one. Ideally is this something that I can achieve in the Splunk Otel helm chart?

Solution:

See details below on processing multi-line logs; this is likely why you are getting a multi-line json showing up as individual log lines

Documentation:

rss.livelink.thread@place:occasion

Kubernetes Observability - 12/10/24

Re: Kubernetes Observability - 12/10/24