Installation

How to setup Splunk Connect on to capture logs from Kubernetes

dzlabs
Engager

I'm trying to setup splunk-connect for kubernetes, I'm currently testing with Splunk Cloud and a k8s running on Docker Desktop.

I did set up an HEC on my splunk could environment and confirmed it can receive events with the generated account, using a curl like this:

curl -k "https://mysplunk.splunkcloud.com:8088/services/collector" \
    -H "Authorization: Splunk MY_HEC_TOKEN" \
    -d '{"event": "Hello, world!", "sourcetype": "manual"}'

Here is my current values file that I'm using to set up Splunk Connect:

 

 

global:
  logLevel: info
  # If local splunk configurations are not present, the global ones will be used (if available)
  splunk:
    # It has exactly the same configs as splunk.hec does
    hec:
      host: mysplunk.splunkcloud.com
      port: 8088
      token: MY_HEC_TOKEN
      protocol: https
      indexName: default
      insecureSSL: false
# local config for logging chart
splunk-kubernetes-logging:
  journalLogPath: /run/log/journal
  splunk:
    hec:
      indexName: k8s-logs
# local config for objects chart
splunk-kubernetes-objects:
  rbac:
    create: true
  serviceAccount:
    create: true
    name: splunk-kubernetes-objects
  kubernetes:
    insecureSSL: true  
  objects:
    core:
      v1:
        - name: pods
        - name: namespaces
        - name: nodes
        - name: services
        - name: config_maps
        - name: secrets
        - name: persistent_volumes
        - name: service_accounts
        - name: persistent_volume_claims
        - name: resource_quotes
        - name: component_statuses
        - name: events
        - name: watch
    apps:
      v1:
        - name: deployments
        - name: daemon_sets 
        - name: replica_sets
        - name: stateful_sets
  splunk:
    hec:
      indexName: k8s-meta
# local config for metrics chart
splunk-kubernetes-metrics:
  rbac:
    create: true
  serviceAccount:
    create: true
    name: splunk-kubernetes-metrics
  splunk:
    hec:
      indexName: k8s-metrics

  kubernetes:
    clusterName: "docker-desktop"
  prometheus_enabled: true

 

The I install Splunk Connect like this

$ helm upgrade splunk-connect-daemonset \
        --set splunk-kubernetes-metrics.splunk.hec.indexName=default \
        https://github.com/splunk/splunk-connect-for-kubernetes/releases/download/1.2.0/splunk-connect-for-kubernetes-1.2.0.tgz

The installation seems to go smooth, I can see the pods created

 

$ kubectl get pods                                                      
NAME                                                              READY   STATUS    RESTARTS   AGE
splunk-connect-daemonset-splunk-kubernetes-logging-zdz5q          1/1     Running   0          23m
splunk-connect-daemonset-splunk-kubernetes-metrics-agg-77cmfx75   1/1     Running   0          23m
splunk-connect-daemonset-splunk-kubernetes-metrics-w2rg6          1/1     Running   0          23m
splunk-connect-daemonset-splunk-kubernetes-objects-5748df8nbl7r   1/1     Running   0          23m

 

Now the problem is no events are being sent to Splunk Cloud account. And look at the logs I can see problems but not sure how to proceed

 

$  k logs splunk-connect-daemonset-splunk-kubernetes-objects-5748df8nbl7r
2020-06-29 20:08:17 +0000 [info]: Worker 0 finished unexpectedly with status 1
2020-06-29 20:08:17 +0000 [info]: adding filter pattern="kube.**" type="jq_transformer"
2020-06-29 20:08:17 +0000 [info]: adding filter pattern="kube.**" type="jq_transformer"
2020-06-29 20:08:18 +0000 [info]: adding match pattern="kube.**" type="splunk_hec"
2020-06-29 20:08:18 +0000 [info]: adding source type="kubernetes_objects"
2020-06-29 20:08:18 +0000 [warn]: #0 both of Plugin @id and path for <storage> are not specified. Using on-memory store.
2020-06-29 20:08:18 +0000 [info]: adding source type="kubernetes_objects"
2020-06-29 20:08:18 +0000 [warn]: #0 both of Plugin @id and path for <storage> are not specified. Using on-memory store.
2020-06-29 20:08:18 +0000 [warn]: parameter 'cluster_name' in <fields>
  cluster_name 
</fields> is not used.
2020-06-29 20:08:18 +0000 [info]: #0 starting fluentd worker pid=58296 ppid=1 worker=0
2020-06-29 20:08:18 +0000 [info]: #0 fluentd worker is now running worker=0
2020-06-29 20:08:18 +0000 [warn]: #0 thread exited by unexpected error plugin=Fluent::Plugin::KubernetesObjectsInput title=:pull_resource_quotes error_class=NoMethodError error="undefined method `get_resource_quotes' for #<Kubeclient::Client:0x000055585882a130>\nDid you mean?  get_resource_quotas\n               get_resource_quota\n               watch_resource_quotas"
#<Thread:0x0000555858d851e8@pull_resource_quotes@/usr/share/gems/gems/fluentd-1.9.1/lib/fluent/plugin_helper/thread.rb:70 run> terminated with exception (report_on_exception is true):
/usr/share/gems/gems/kubeclient-4.6.0/lib/kubeclient/common.rb:103:in `method_missing': undefined method `get_resource_quotes' for #<Kubeclient::Client:0x000055585882a130> (NoMethodError)
Did you mean?  get_resource_quotas
               get_resource_quota
               watch_resource_quotas
	from /usr/share/gems/gems/kubeclient-4.6.0/lib/kubeclient/common.rb:101:in `method_missing'
	from /opt/app-root/src/gem/fluent-plugin-kubernetes-objects-1.1.3/lib/fluent/plugin/in_kubernetes_objects.rb:203:in `public_send'
	from /opt/app-root/src/gem/fluent-plugin-kubernetes-objects-1.1.3/lib/fluent/plugin/in_kubernetes_objects.rb:203:in `block in create_pull_thread'
	from /usr/share/gems/gems/fluentd-1.9.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2020-06-29 20:08:18 +0000 [error]: #0 unexpected error error_class=NoMethodError error="undefined method `get_resource_quotes' for #<Kubeclient::Client:0x000055585882a130>\nDid you mean?  get_resource_quotas\n               get_resource_quota\n               watch_resource_quotas"
  2020-06-29 20:08:18 +0000 [error]: #0 /usr/share/gems/gems/kubeclient-4.6.0/lib/kubeclient/common.rb:103:in `method_missing'
  2020-06-29 20:08:18 +0000 [error]: #0 /usr/share/gems/gems/kubeclient-4.6.0/lib/kubeclient/common.rb:101:in `method_missing'
  2020-06-29 20:08:18 +0000 [error]: #0 /opt/app-root/src/gem/fluent-plugin-kubernetes-objects-1.1.3/lib/fluent/plugin/in_kubernetes_objects.rb:203:in `public_send'
  2020-06-29 20:08:18 +0000 [error]: #0 /opt/app-root/src/gem/fluent-plugin-kubernetes-objects-1.1.3/lib/fluent/plugin/in_kubernetes_objects.rb:203:in `block in create_pull_thread'
  2020-06-29 20:08:18 +0000 [error]: #0 /usr/share/gems/gems/fluentd-1.9.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2020-06-29 20:08:18 +0000 [error]: #0 unexpected error error_class=NoMethodError error="undefined method `get_resource_quotes' for #<Kubeclient::Client:0x000055585882a130>\nDid you mean?  get_resource_quotas\n               get_resource_quota\n               watch_resource_quotas"
  2020-06-29 20:08:18 +0000 [error]: #0 suppressed same stacktrace
2020-06-29 20:08:19 +0000 [info]: Worker 0 finished unexpectedly with status 1

 

And the logs from the logging daemon set

 

$ k logs splunk-connect-daemonset-splunk-kubernetes-logging-zdz5q
2020-06-29 20:09:45 +0000 [info]: #0 Timeout flush: tail.containers.var.log.containers.kube-controller-manager-docker-desktop_kube-system_kube-controller-manager-bcf1f05eb5c2c0ede7bcebe0934cbe3ba246937f7b623871627520c76f287498.log:stderr
2020-06-29 20:09:48 +0000 [error]: #0 failed to flush the buffer, and hit limit for retries. dropping all chunks in the buffer queue. retry_times=3 records=130227 error_class=OpenSSL::SSL::SSLError error="SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)"
  2020-06-29 20:09:48 +0000 [error]: #0 suppressed same stacktrace
2020-06-29 20:09:49 +0000 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2020-06-29 20:09:50 +0000 chunk="5a93ea3871f0b809090979783479a275" error_class=OpenSSL::SSL::SSLError error="SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)"
  2020-06-29 20:09:49 +0000 [warn]: #0 suppressed same stacktrace
2020-06-29 20:09:50 +0000 [warn]: #0 failed to flush the buffer. retry_time=1 next_retry_seconds=2020-06-29 20:09:51 +0000 chunk="5a93ea3871f0b809090979783479a275" error_class=OpenSSL::SSL::SSLError error="SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)"
  2020-06-29 20:09:50 +0000 [warn]: #0 suppressed same stacktrace
2020-06-29 20:09:51 +0000 [warn]: #0 failed to flush the buffer. retry_time=2 next_retry_seconds=2020-06-29 20:09:53 +0000 chunk="5a93ea3871f0b809090979783479a275" error_class=OpenSSL::SSL::SSLError error="SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)"
  2020-06-29 20:09:51 +0000 [warn]: #0 suppressed same stacktrace
2020-06-29 20:09:53 +0000 [warn]: #0 failed to flush the buffer. retry_time=3 next_retry_seconds=2020-06-29 20:09:57 +0000 chunk="5a93ea3871f0b809090979783479a275" error_class=OpenSSL::SSL::SSLError error="SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)"
  2020-06-29 20:09:53 +0000 [warn]: #0 suppressed same stacktrace

 

 

Labels (1)
Tags (1)
0 Karma
Get Updates on the Splunk Community!

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...