We've started using the Splunk Connect for Kubernetes. We have a heavy forwarder setup that hosts the HEC endpoint, and forwards those logs onto the indexers. We had an issue where the indexers went down, no indication that the HEC endpoint was down though. During this time our logs from these K8 clusters are missing. Other systems that use the normal universal forwarder ended up populating the logs during down time, however the logs from those clusters did not.
I'm not familiar with K8 and wasn't involved in the deployment of the Splunk Connect pieces. I'm assuming that when using an HEC, if the logs don't make it into splunk, they aren't retried or cached anywhere? Or is that some sort of setting we may have missed?
Thanks for looking,
Jeremiah
