I'm trying to setup splunk-connect for kubernetes, I'm currently testing with Splunk Cloud and a k8s running on Docker Desktop.
I did set up an HEC on my splunk could environment and confirmed it can receive events with the generated account, using a curl like this:
curl -k "https://mysplunk.splunkcloud.com:8088/services/collector" \
-H "Authorization: Splunk MY_HEC_TOKEN" \
-d '{"event": "Hello, world!", "sourcetype": "manual"}'
Here is my current values file that I'm using to set up Splunk Connect:
global:
logLevel: info
# If local splunk configurations are not present, the global ones will be used (if available)
splunk:
# It has exactly the same configs as splunk.hec does
hec:
host: mysplunk.splunkcloud.com
port: 8088
token: MY_HEC_TOKEN
protocol: https
indexName: default
insecureSSL: false
# local config for logging chart
splunk-kubernetes-logging:
journalLogPath: /run/log/journal
splunk:
hec:
indexName: k8s-logs
# local config for objects chart
splunk-kubernetes-objects:
rbac:
create: true
serviceAccount:
create: true
name: splunk-kubernetes-objects
kubernetes:
insecureSSL: true
objects:
core:
v1:
- name: pods
- name: namespaces
- name: nodes
- name: services
- name: config_maps
- name: secrets
- name: persistent_volumes
- name: service_accounts
- name: persistent_volume_claims
- name: resource_quotes
- name: component_statuses
- name: events
- name: watch
apps:
v1:
- name: deployments
- name: daemon_sets
- name: replica_sets
- name: stateful_sets
splunk:
hec:
indexName: k8s-meta
# local config for metrics chart
splunk-kubernetes-metrics:
rbac:
create: true
serviceAccount:
create: true
name: splunk-kubernetes-metrics
splunk:
hec:
indexName: k8s-metrics
kubernetes:
clusterName: "docker-desktop"
prometheus_enabled: true
The I install Splunk Connect like this
$ helm upgrade splunk-connect-daemonset \
--set splunk-kubernetes-metrics.splunk.hec.indexName=default \
https://github.com/splunk/splunk-connect-for-kubernetes/releases/download/1.2.0/splunk-connect-for-kubernetes-1.2.0.tgz
The installation seems to go smooth, I can see the pods created
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
splunk-connect-daemonset-splunk-kubernetes-logging-zdz5q 1/1 Running 0 23m
splunk-connect-daemonset-splunk-kubernetes-metrics-agg-77cmfx75 1/1 Running 0 23m
splunk-connect-daemonset-splunk-kubernetes-metrics-w2rg6 1/1 Running 0 23m
splunk-connect-daemonset-splunk-kubernetes-objects-5748df8nbl7r 1/1 Running 0 23m
Now the problem is no events are being sent to Splunk Cloud account. And look at the logs I can see problems but not sure how to proceed
$ k logs splunk-connect-daemonset-splunk-kubernetes-objects-5748df8nbl7r
2020-06-29 20:08:17 +0000 [info]: Worker 0 finished unexpectedly with status 1
2020-06-29 20:08:17 +0000 [info]: adding filter pattern="kube.**" type="jq_transformer"
2020-06-29 20:08:17 +0000 [info]: adding filter pattern="kube.**" type="jq_transformer"
2020-06-29 20:08:18 +0000 [info]: adding match pattern="kube.**" type="splunk_hec"
2020-06-29 20:08:18 +0000 [info]: adding source type="kubernetes_objects"
2020-06-29 20:08:18 +0000 [warn]: #0 both of Plugin @id and path for <storage> are not specified. Using on-memory store.
2020-06-29 20:08:18 +0000 [info]: adding source type="kubernetes_objects"
2020-06-29 20:08:18 +0000 [warn]: #0 both of Plugin @id and path for <storage> are not specified. Using on-memory store.
2020-06-29 20:08:18 +0000 [warn]: parameter 'cluster_name' in <fields>
cluster_name
</fields> is not used.
2020-06-29 20:08:18 +0000 [info]: #0 starting fluentd worker pid=58296 ppid=1 worker=0
2020-06-29 20:08:18 +0000 [info]: #0 fluentd worker is now running worker=0
2020-06-29 20:08:18 +0000 [warn]: #0 thread exited by unexpected error plugin=Fluent::Plugin::KubernetesObjectsInput title=:pull_resource_quotes error_class=NoMethodError error="undefined method `get_resource_quotes' for #<Kubeclient::Client:0x000055585882a130>\nDid you mean? get_resource_quotas\n get_resource_quota\n watch_resource_quotas"
#<Thread:0x0000555858d851e8@pull_resource_quotes@/usr/share/gems/gems/fluentd-1.9.1/lib/fluent/plugin_helper/thread.rb:70 run> terminated with exception (report_on_exception is true):
/usr/share/gems/gems/kubeclient-4.6.0/lib/kubeclient/common.rb:103:in `method_missing': undefined method `get_resource_quotes' for #<Kubeclient::Client:0x000055585882a130> (NoMethodError)
Did you mean? get_resource_quotas
get_resource_quota
watch_resource_quotas
from /usr/share/gems/gems/kubeclient-4.6.0/lib/kubeclient/common.rb:101:in `method_missing'
from /opt/app-root/src/gem/fluent-plugin-kubernetes-objects-1.1.3/lib/fluent/plugin/in_kubernetes_objects.rb:203:in `public_send'
from /opt/app-root/src/gem/fluent-plugin-kubernetes-objects-1.1.3/lib/fluent/plugin/in_kubernetes_objects.rb:203:in `block in create_pull_thread'
from /usr/share/gems/gems/fluentd-1.9.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2020-06-29 20:08:18 +0000 [error]: #0 unexpected error error_class=NoMethodError error="undefined method `get_resource_quotes' for #<Kubeclient::Client:0x000055585882a130>\nDid you mean? get_resource_quotas\n get_resource_quota\n watch_resource_quotas"
2020-06-29 20:08:18 +0000 [error]: #0 /usr/share/gems/gems/kubeclient-4.6.0/lib/kubeclient/common.rb:103:in `method_missing'
2020-06-29 20:08:18 +0000 [error]: #0 /usr/share/gems/gems/kubeclient-4.6.0/lib/kubeclient/common.rb:101:in `method_missing'
2020-06-29 20:08:18 +0000 [error]: #0 /opt/app-root/src/gem/fluent-plugin-kubernetes-objects-1.1.3/lib/fluent/plugin/in_kubernetes_objects.rb:203:in `public_send'
2020-06-29 20:08:18 +0000 [error]: #0 /opt/app-root/src/gem/fluent-plugin-kubernetes-objects-1.1.3/lib/fluent/plugin/in_kubernetes_objects.rb:203:in `block in create_pull_thread'
2020-06-29 20:08:18 +0000 [error]: #0 /usr/share/gems/gems/fluentd-1.9.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2020-06-29 20:08:18 +0000 [error]: #0 unexpected error error_class=NoMethodError error="undefined method `get_resource_quotes' for #<Kubeclient::Client:0x000055585882a130>\nDid you mean? get_resource_quotas\n get_resource_quota\n watch_resource_quotas"
2020-06-29 20:08:18 +0000 [error]: #0 suppressed same stacktrace
2020-06-29 20:08:19 +0000 [info]: Worker 0 finished unexpectedly with status 1
And the logs from the logging daemon set
$ k logs splunk-connect-daemonset-splunk-kubernetes-logging-zdz5q
2020-06-29 20:09:45 +0000 [info]: #0 Timeout flush: tail.containers.var.log.containers.kube-controller-manager-docker-desktop_kube-system_kube-controller-manager-bcf1f05eb5c2c0ede7bcebe0934cbe3ba246937f7b623871627520c76f287498.log:stderr
2020-06-29 20:09:48 +0000 [error]: #0 failed to flush the buffer, and hit limit for retries. dropping all chunks in the buffer queue. retry_times=3 records=130227 error_class=OpenSSL::SSL::SSLError error="SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)"
2020-06-29 20:09:48 +0000 [error]: #0 suppressed same stacktrace
2020-06-29 20:09:49 +0000 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2020-06-29 20:09:50 +0000 chunk="5a93ea3871f0b809090979783479a275" error_class=OpenSSL::SSL::SSLError error="SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)"
2020-06-29 20:09:49 +0000 [warn]: #0 suppressed same stacktrace
2020-06-29 20:09:50 +0000 [warn]: #0 failed to flush the buffer. retry_time=1 next_retry_seconds=2020-06-29 20:09:51 +0000 chunk="5a93ea3871f0b809090979783479a275" error_class=OpenSSL::SSL::SSLError error="SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)"
2020-06-29 20:09:50 +0000 [warn]: #0 suppressed same stacktrace
2020-06-29 20:09:51 +0000 [warn]: #0 failed to flush the buffer. retry_time=2 next_retry_seconds=2020-06-29 20:09:53 +0000 chunk="5a93ea3871f0b809090979783479a275" error_class=OpenSSL::SSL::SSLError error="SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)"
2020-06-29 20:09:51 +0000 [warn]: #0 suppressed same stacktrace
2020-06-29 20:09:53 +0000 [warn]: #0 failed to flush the buffer. retry_time=3 next_retry_seconds=2020-06-29 20:09:57 +0000 chunk="5a93ea3871f0b809090979783479a275" error_class=OpenSSL::SSL::SSLError error="SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)"
2020-06-29 20:09:53 +0000 [warn]: #0 suppressed same stacktrace