How to setup Splunk Connect on to capture logs from Kubernetes

[dzlabs]

I'm trying to setup splunk-connect for kubernetes, I'm currently testing with Splunk Cloud and a k8s running on Docker Desktop.

I did set up an HEC on my splunk could environment and confirmed it can receive events with the generated account, using a curl like this:

curl -k "https://mysplunk.splunkcloud.com:8088/services/collector" \
    -H "Authorization: Splunk MY_HEC_TOKEN" \
    -d '{"event": "Hello, world!", "sourcetype": "manual"}'

Here is my current values file that I'm using to set up Splunk Connect:

global:
  logLevel: info
  # If local splunk configurations are not present, the global ones will be used (if available)
  splunk:
    # It has exactly the same configs as splunk.hec does
    hec:
      host: mysplunk.splunkcloud.com
      port: 8088
      token: MY_HEC_TOKEN
      protocol: https
      indexName: default
      insecureSSL: false
# local config for logging chart
splunk-kubernetes-logging:
  journalLogPath: /run/log/journal
  splunk:
    hec:
      indexName: k8s-logs
# local config for objects chart
splunk-kubernetes-objects:
  rbac:
    create: true
  serviceAccount:
    create: true
    name: splunk-kubernetes-objects
  kubernetes:
    insecureSSL: true  
  objects:
    core:
      v1:
        - name: pods
        - name: namespaces
        - name: nodes
        - name: services
        - name: config_maps
        - name: secrets
        - name: persistent_volumes
        - name: service_accounts
        - name: persistent_volume_claims
        - name: resource_quotes
        - name: component_statuses
        - name: events
        - name: watch
    apps:
      v1:
        - name: deployments
        - name: daemon_sets 
        - name: replica_sets
        - name: stateful_sets
  splunk:
    hec:
      indexName: k8s-meta
# local config for metrics chart
splunk-kubernetes-metrics:
  rbac:
    create: true
  serviceAccount:
    create: true
    name: splunk-kubernetes-metrics
  splunk:
    hec:
      indexName: k8s-metrics

  kubernetes:
    clusterName: "docker-desktop"
  prometheus_enabled: true

The I install Splunk Connect like this

$ helm upgrade splunk-connect-daemonset \
        --set splunk-kubernetes-metrics.splunk.hec.indexName=default \
        https://github.com/splunk/splunk-connect-for-kubernetes/releases/download/1.2.0/splunk-connect-for-kubernetes-1.2.0.tgz

The installation seems to go smooth, I can see the pods created

$ kubectl get pods                                                      
NAME                                                              READY   STATUS    RESTARTS   AGE
splunk-connect-daemonset-splunk-kubernetes-logging-zdz5q          1/1     Running   0          23m
splunk-connect-daemonset-splunk-kubernetes-metrics-agg-77cmfx75   1/1     Running   0          23m
splunk-connect-daemonset-splunk-kubernetes-metrics-w2rg6          1/1     Running   0          23m
splunk-connect-daemonset-splunk-kubernetes-objects-5748df8nbl7r   1/1     Running   0          23m

Now the problem is no events are being sent to Splunk Cloud account. And look at the logs I can see problems but not sure how to proceed

$  k logs splunk-connect-daemonset-splunk-kubernetes-objects-5748df8nbl7r
2020-06-29 20:08:17 +0000 [info]: Worker 0 finished unexpectedly with status 1
2020-06-29 20:08:17 +0000 [info]: adding filter pattern="kube.**" type="jq_transformer"
2020-06-29 20:08:17 +0000 [info]: adding filter pattern="kube.**" type="jq_transformer"
2020-06-29 20:08:18 +0000 [info]: adding match pattern="kube.**" type="splunk_hec"
2020-06-29 20:08:18 +0000 [info]: adding source type="kubernetes_objects"
2020-06-29 20:08:18 +0000 [warn]: #0 both of Plugin @id and path for <storage> are not specified. Using on-memory store.
2020-06-29 20:08:18 +0000 [info]: adding source type="kubernetes_objects"
2020-06-29 20:08:18 +0000 [warn]: #0 both of Plugin @id and path for <storage> are not specified. Using on-memory store.
2020-06-29 20:08:18 +0000 [warn]: parameter 'cluster_name' in <fields>
  cluster_name 
</fields> is not used.
2020-06-29 20:08:18 +0000 [info]: #0 starting fluentd worker pid=58296 ppid=1 worker=0
2020-06-29 20:08:18 +0000 [info]: #0 fluentd worker is now running worker=0
2020-06-29 20:08:18 +0000 [warn]: #0 thread exited by unexpected error plugin=Fluent::Plugin::KubernetesObjectsInput title=:pull_resource_quotes error_class=NoMethodError error="undefined method `get_resource_quotes' for #<Kubeclient::Client:0x000055585882a130>\nDid you mean?  get_resource_quotas\n               get_resource_quota\n               watch_resource_quotas"
#<Thread:0x0000555858d851e8@pull_resource_quotes@/usr/share/gems/gems/fluentd-1.9.1/lib/fluent/plugin_helper/thread.rb:70 run> terminated with exception (report_on_exception is true):
/usr/share/gems/gems/kubeclient-4.6.0/lib/kubeclient/common.rb:103:in `method_missing': undefined method `get_resource_quotes' for #<Kubeclient::Client:0x000055585882a130> (NoMethodError)
Did you mean?  get_resource_quotas
               get_resource_quota
               watch_resource_quotas
	from /usr/share/gems/gems/kubeclient-4.6.0/lib/kubeclient/common.rb:101:in `method_missing'
	from /opt/app-root/src/gem/fluent-plugin-kubernetes-objects-1.1.3/lib/fluent/plugin/in_kubernetes_objects.rb:203:in `public_send'
	from /opt/app-root/src/gem/fluent-plugin-kubernetes-objects-1.1.3/lib/fluent/plugin/in_kubernetes_objects.rb:203:in `block in create_pull_thread'
	from /usr/share/gems/gems/fluentd-1.9.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2020-06-29 20:08:18 +0000 [error]: #0 unexpected error error_class=NoMethodError error="undefined method `get_resource_quotes' for #<Kubeclient::Client:0x000055585882a130>\nDid you mean?  get_resource_quotas\n               get_resource_quota\n               watch_resource_quotas"
  2020-06-29 20:08:18 +0000 [error]: #0 /usr/share/gems/gems/kubeclient-4.6.0/lib/kubeclient/common.rb:103:in `method_missing'
  2020-06-29 20:08:18 +0000 [error]: #0 /usr/share/gems/gems/kubeclient-4.6.0/lib/kubeclient/common.rb:101:in `method_missing'
  2020-06-29 20:08:18 +0000 [error]: #0 /opt/app-root/src/gem/fluent-plugin-kubernetes-objects-1.1.3/lib/fluent/plugin/in_kubernetes_objects.rb:203:in `public_send'
  2020-06-29 20:08:18 +0000 [error]: #0 /opt/app-root/src/gem/fluent-plugin-kubernetes-objects-1.1.3/lib/fluent/plugin/in_kubernetes_objects.rb:203:in `block in create_pull_thread'
  2020-06-29 20:08:18 +0000 [error]: #0 /usr/share/gems/gems/fluentd-1.9.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2020-06-29 20:08:18 +0000 [error]: #0 unexpected error error_class=NoMethodError error="undefined method `get_resource_quotes' for #<Kubeclient::Client:0x000055585882a130>\nDid you mean?  get_resource_quotas\n               get_resource_quota\n               watch_resource_quotas"
  2020-06-29 20:08:18 +0000 [error]: #0 suppressed same stacktrace
2020-06-29 20:08:19 +0000 [info]: Worker 0 finished unexpectedly with status 1

And the logs from the logging daemon set

$ k logs splunk-connect-daemonset-splunk-kubernetes-logging-zdz5q
2020-06-29 20:09:45 +0000 [info]: #0 Timeout flush: tail.containers.var.log.containers.kube-controller-manager-docker-desktop_kube-system_kube-controller-manager-bcf1f05eb5c2c0ede7bcebe0934cbe3ba246937f7b623871627520c76f287498.log:stderr
2020-06-29 20:09:48 +0000 [error]: #0 failed to flush the buffer, and hit limit for retries. dropping all chunks in the buffer queue. retry_times=3 records=130227 error_class=OpenSSL::SSL::SSLError error="SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)"
  2020-06-29 20:09:48 +0000 [error]: #0 suppressed same stacktrace
2020-06-29 20:09:49 +0000 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2020-06-29 20:09:50 +0000 chunk="5a93ea3871f0b809090979783479a275" error_class=OpenSSL::SSL::SSLError error="SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)"
  2020-06-29 20:09:49 +0000 [warn]: #0 suppressed same stacktrace
2020-06-29 20:09:50 +0000 [warn]: #0 failed to flush the buffer. retry_time=1 next_retry_seconds=2020-06-29 20:09:51 +0000 chunk="5a93ea3871f0b809090979783479a275" error_class=OpenSSL::SSL::SSLError error="SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)"
  2020-06-29 20:09:50 +0000 [warn]: #0 suppressed same stacktrace
2020-06-29 20:09:51 +0000 [warn]: #0 failed to flush the buffer. retry_time=2 next_retry_seconds=2020-06-29 20:09:53 +0000 chunk="5a93ea3871f0b809090979783479a275" error_class=OpenSSL::SSL::SSLError error="SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)"
  2020-06-29 20:09:51 +0000 [warn]: #0 suppressed same stacktrace
2020-06-29 20:09:53 +0000 [warn]: #0 failed to flush the buffer. retry_time=3 next_retry_seconds=2020-06-29 20:09:57 +0000 chunk="5a93ea3871f0b809090979783479a275" error_class=OpenSSL::SSL::SSLError error="SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)"
  2020-06-29 20:09:53 +0000 [warn]: #0 suppressed same stacktrace