topic Re: Are any Fluentd apps Splunk vetted/supported? Or is there a preferred cloud-native solution for logging Kubernetes logs? in Getting Data In

Are any Fluentd apps Splunk vetted/supported? Or is there a preferred cloud-native solution for logging Kubernetes logs?

mcluver — Tue, 03 Oct 2017 17:54:25 GMT

We’re looking to get our Kubernetes logs into Splunk and it appears the best (most cloud native) way to do that is to forward the logs from Fluentd to Splunk HEC (HTTP Event Collector). With that being said, we see where there are a number of plugins that people have developed for Fluentd for this use-case, see: Fluentd Plugins Could you guys please tell us if any of these were developed by Splunk employees or are officially vetted/supported?

Does Splunk have another cloud native solution that they recommend instead? Don’t say the UF (Splunk Universal Forwarder). I also found this Splunk Answers post regarding the same topic for a bit of background on what others were doing cloud natively. Thanks for any assistance with this question.

Thanks & Best regards,
Matt

Re: Are any Fluentd apps Splunk vetted/supported? Or is there a preferred cloud-native solution for logging Kubernetes logs?

sduff_splunk — Tue, 03 Oct 2017 23:36:24 GMT

Why are you dismissing the Universal Forwarder? It is as "cloud native" as anything else you might find?

Re: Are any Fluentd apps Splunk vetted/supported? Or is there a preferred cloud-native solution for logging Kubernetes logs?

mcluver — Wed, 04 Oct 2017 00:32:56 GMT

We feel the Splunk Forwarder is more for host/node level data collection and that's not the way we were planning to log our Kubernetes infrastructure. If one were to want to write all logs back to the host/node level, then yes the Splunk Forwarder would work fine and we use it in the cloud at that level significantly already for many other workloads. In our opinion, the Splunk HEC is much more well suited to the task of collecting logs from something like Kubernetes/Docker which should be more directly from the logging driver or container engine level.

The official k8s logging documentation mentions several different logging approaches, with node-level and cluster-level being the two main parent categories. Cluster-level is the one that we believe is the better approach and why we're looking for a different solution than the Splunk Forwarder in this space.

Re: Are any Fluentd apps Splunk vetted/supported? Or is there a preferred cloud-native solution for logging Kubernetes logs?

outcoldman — Wed, 04 Oct 2017 03:20:42 GMT

One solution to monitor Kubernetes and collect all logs is to use our collector

SplunkBase: https://splunkbase.splunk.com/app/3743/
Youtube Demo https://www.youtube.com/watch?v=C2zOO2XX5TI
Installation/configuration instructions https://github.com/outcoldsolutions/collector/tree/master/kubernetes

With provided configuration it will automatically pick up all logs, enrich them with kubernetes metadata and ship it to Splunk.

Let me know if you will have any questions.

Edited (2017-10-05): posted link to published application on splunkbase

Re: Are any Fluentd apps Splunk vetted/supported? Or is there a preferred cloud-native solution for logging Kubernetes logs?

mcluver — Thu, 05 Oct 2017 16:23:53 GMT

That looks like a really nice solution for Kubernetes logging and metrics. Probably the best solution to date for Splunk I've seen. Is Splunk not planning on releasing their own TA for this platform at some point?

Being a Splunk Enterprise Security customer I think we'd be interesting in the collection being CIM compliant as well.

Re: Are any Fluentd apps Splunk vetted/supported? Or is there a preferred cloud-native solution for logging Kubernetes logs?

outcoldman — Thu, 05 Oct 2017 16:46:12 GMT

CIM compliant is on our radar for "Monitoring Kubernetes" and "Collector for Kubernetes".

Re: Are any Fluentd apps Splunk vetted/supported? Or is there a preferred cloud-native solution for logging Kubernetes logs?

epeterfi_splunk — Wed, 04 Jul 2018 12:41:47 GMT

To monitor Kubernetes please go here and follow the steps: https://github.com/splunk/splunk-connect-for-kubernetes

Re: Are any Fluentd apps Splunk vetted/supported? Or is there a preferred cloud-native solution for logging Kubernetes logs?

epeterfi_splunk — Fri, 05 Oct 2018 11:12:08 GMT

See below my comment. Connect for K8 was launched months ago and works great. Also comes with an app.