topic Re: K8s Pod Name is not available in metrics/alerting, making it impossible to add pod name to alert templates in Splunk AppDynamics

K8s Pod Name is not available in metrics/alerting, making it impossible to add pod name to alert templates

John_Groetzinge — Wed, 27 Sep 2023 15:03:54 GMT

We have set up cluster monitoring for K8s cluster to monitor when pods get killed, failed, etc. The Alert we get looks like the following:

We have hundreds of pods and namespaces in the cluster and I would like the alert summary to contain namespace and pod name, otherwise I don't know if it's something I can ignore or not. For example is someone is in a dev namespace testing a new app the pod alerts might go off a lot and there is not way to know if this is a production namespace/pod without going into the app. And When I wake up in the morning with 100 email alerts for pods failing i don't know if prod id falling over or someone set up a new namespace to test...

It doesn't seem to be possible to get the pod name, even though it's available in the dashboard view of the event:

All I want is Email/HTML template to include the namespace and Pod name, but that doesn't seem to be possible today.

Re: K8s Pod Name is not available in metrics/alerting, making it impossible to add pod name to alert templates

iamryan — Wed, 27 Sep 2023 18:23:52 GMT

Hello @John.Groetzinger,

I saw a similar post as this was also created that read more like a feature request. Thanks for providing more detail here.

Are you looking for workarounds or do you want to submit this as a feature request?

Re: K8s Pod Name is not available in metrics/alerting, making it impossible to add pod name to alert templates

John_Groetzinge — Wed, 27 Sep 2023 18:32:01 GMT

This is meant to be a feature request as I was told it was not possible to do this. If it is possible to do today that would be great, I don't have admin role on my tenant, the admins told me I needed to open an idea/feature request. If this is possible today then I would just like to know how to do it, no one i've talked to seems to think it is possible and hence asked me to open the discussion here.

Re: K8s Pod Name is not available in metrics/alerting, making it impossible to add pod name to alert templates

iamryan — Wed, 27 Sep 2023 18:40:11 GMT

Hi @John.Groetzinger,

Thanks for clarifying! You can head on over to the Idea Exchange and do that.
Please have a read of the Idea Exchange Submission Guidelines to make sure you submit a complete idea.