All Apps and Add-ons

Splunk Connect for Kubernetes - splunk-kubernetes-objects - error_class=Net::OpenTimeout error="execution expired"

rfrodrigues
New Member

I’m experiencing a problem when trying to use the SCK agent for collecting Kubernetes objects data (splunk-kubernetes-objects).
The SCK agent throws an error “execution expired” whenever it tries to post the data using HEC. See the log excerpts below:

2019-08-13 21:47:37 +0000 [debug]: #0 Sending 4547 bytes to Splunk.
2019-08-13 21:47:37 +0000 [trace]: #0 POST https://138.85.114.182:8088/services/collector body={"host":"","time":"1565732853.4252117","index":"vecc_csa_kubernetes_logs","source":"namespace:splunk/pod:splunk-connect-splunk-kubernetes-objects-6cb5876855-rmqpf","sourcetype":"kube:objects:pods","fields":{"cluster_name":"cluster_name"},"event":"{\"metadata\":{\"name\":\"splunk-connect-splunk-kubernetes-objects-6cb5876855-rmqpf\",\"generateName\":\"splunk-connect-splunk-kubernetes-objects-6cb5876855-\",\"namespace\":\"splunk\",\"selfLink\":\"/api/v1/namespaces/splunk/pods/splunk-connect-splunk-kubernetes-objects-6cb5876855-rmqpf\",\"uid\":\"f3dc7102-be13-11e9-8dde-fa163e5b4bad\",\"resourceVersion\":\"5401185\",\"creationTimestamp\":\"2019-08-13T21:47:31Z\",\"labels\":{\"app\":\"splunk-kubernetes-objects\",\"engine\":\"fluentd\",\"pod-template-hash\":\"6cb5876855\",\"release\":\"splunk-connect\"},\"annotations\":{\"checksum/config\":\"b14e319dbcdaddd52e542e9fe9b529381b0f918591820bcdfc4a10fd2b572fff\"},\"ownerReferences\":[{\"apiVersion\":\"apps/v1\",\"kind\":\"ReplicaSet\",\"name\":\"splunk-connect-splunk-kubernetes-objects-6cb5876855\",\"uid\":\"f3da97df-be13-11e9-8dde-fa163e5b4bad\",\"controller\":true,\"blockOwnerDeletion\":true}],\"requestResourceVersion\":\"5401187\"},\"spec\":{\"volumes\":[{\"name\":\"conf-configmap\",\"configMap\":{\"name\":\"splunk-connect-splunk-kubernetes-objects\",\"defaultMode\":420}},{\"name\":\"secrets\",\"secret\":{\"secretName\":\"splunk-kubernetes-objects\",\"defaultMode\":420}},{\"name\":\"splunk-connect-splunk-kubernetes-objects-token-4m9w6\",\"secret\":{\"secretName\":\"splunk-connect-splunk-kubernetes-objects-token-4m9w6\",\"defaultMode\":420}}],\"containers\":[{\"name\":\"splunk-fluentd-k8s-objects\",\"image\":\"k8s-registry.eccd.local/splunk/kube-objects:1.1.0\",\"args\":[\"-c\",\"/fluentd/etc/fluent.conf\"],\"env\":[{\"name\":\"MY_NAMESPACE\",\"valueFrom\":{\"fieldRef\":{\"apiVersion\":\"v1\",\"fieldPath\":\"metadata.namespace\"}}},{\"name\":\"MY_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"apiVersion\":\"v1\",\"fieldPath\":\"metadata.name\"}}},{\"name\":\"NODE_NAME\",\"valueFrom\":{\"fieldRef\":{\"apiVersion\":\"v1\",\"fieldPath\":\"spec.nodeName\"}}},{\"name\":\"SPLUNK_HEC_TOKEN\",\"valueFrom\":{\"secretKeyRef\":{\"name\":\"splunk-kubernetes-objects\",\"key\":\"splunk_hec_token\"}}}],\"resources\":{\"requests\":{\"cpu\":\"100m\",\"memory\":\"200Mi\"}},\"volumeMounts\":[{\"name\":\"conf-configmap\",\"mountPath\":\"/fluentd/etc\"},{\"name\":\"secrets\",\"readOnly\":true,\"mountPath\":\"/fluentd/etc/splunk\"},{\"name\":\"splunk-connect-splunk-kubernetes-objects-token-4m9w6\",\"readOnly\":true,\"mountPath\":\"/var/run/secrets/kubernetes.io/serviceaccount\"}],\"terminationMessagePath\":\"/dev/termination-log\",\"terminationMessagePolicy\":\"File\",\"imagePullPolicy\":\"IfNotPresent\"}],\"restartPolicy\":\"Always\",\"terminationGracePeriodSeconds\":30,\"dnsPolicy\":\"ClusterFirst\",\"serviceAccountName\":\"splunk-connect-splunk-kubernetes-objects\",\"serviceAccount\":\"splunk-connect-splunk-kubernetes-objects\",\"nodeName\":\"worker-pool1-xmbq8dza-csa-eccd-v2\",\"securityContext\":{},\"schedulerName\":\"default-scheduler\",\"tolerations\":[{\"key\":\"node.kubernetes.io/not-ready\",\"operator\":\"Exists\",\"effect\":\"NoExecute\",\"tolerationSeconds\":300},{\"key\":\"node.kubernetes.io/unreachable\",\"operator\":\"Exists\",\"effect\":\"NoExecute\",\"tolerationSeconds\":300}],\"priority\":0,\"enableServiceLinks\":true},\"status\":{\"phase\":\"Running\",\"conditions\":[{\"type\":\"Initialized\",\"status\":\"True\",\"lastProbeTime\":null,\"lastTransitionTime\":\"2019-08-13T21:47:31Z\"},{\"type\":\"Ready\",\"status\":\"True\",\"lastProbeTime\":null,\"lastTransitionTime\":\"2019-08-13T21:47:32Z\"},{\"type\":\"ContainersReady\",\"status\":\"True\",\"lastProbeTime\":null,\"lastTransitionTime\":\"2019-08-13T21:47:32Z\"},{\"type\":\"PodScheduled\",\"status\":\"True\",\"lastProbeTime\":null,\"lastTransitionTime\":\"2019-08-13T21:47:31Z\"}],\"hostIP\":\"10.0.10.2\",\"podIP\":\"192.168.20.23\",\"startTime\":\"2019-08-13T21:47:31Z\",\"containerStatuses\":[{\"name\":\"splunk-fluentd-k8s-objects\",\"state\":{\"running\":{\"startedAt\":\"2019-08-13T21:47:32Z\"}},\"lastState\":{},\"ready\":true,\"restartCount\":0,\"image\":\"k8s-registry.eccd.local/splunk/kube-objects:1.1.0\",\"imageID\":\"docker-pullable://k8s-registry.eccd.local/splunk/kube-objects@sha256:209ea3157615eec0bd43bcae5450c4a7e2c962ce5c8dd42e890ca81dfc551103\",\"containerID\":\"docker://4c59e1927a3d386d8707e86033c73e9a82a7e56823811572abefffdaceda5e79\"}],\"qosClass\":\"Burstable\"}}"}

2019-08-13 21:48:37 +0000 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2019-08-13 21:48:38 +0000 chunk="59006968aad29c0ccc8b3c2c2ee28461" error_class=Net::OpenTimeout error="execution expired"
2019-08-13 21:48:37 +0000 [warn]: #0 /usr/local/lib/ruby/2.5.0/net/http.rb:937:in initialize'
2019-08-13 21:48:37 +0000 [warn]: #0 /usr/local/lib/ruby/2.5.0/net/http.rb:937:in
open'
2019-08-13 21:48:37 +0000 [warn]: #0 /usr/local/lib/ruby/2.5.0/net/http.rb:937:in block in connect'
2019-08-13 21:48:37 +0000 [warn]: #0 /usr/local/lib/ruby/2.5.0/timeout.rb:103:in
timeout'
2019-08-13 21:48:37 +0000 [warn]: #0 /usr/local/lib/ruby/2.5.0/net/http.rb:935:in connect'
2019-08-13 21:48:37 +0000 [warn]: #0 /usr/local/lib/ruby/2.5.0/net/http.rb:920:in
do_start'
2019-08-13 21:48:37 +0000 [warn]: #0 /usr/local/lib/ruby/2.5.0/net/http.rb:915:in start'
2019-08-13 21:48:37 +0000 [warn]: #0 /usr/local/bundle/gems/net-http-persistent-3.0.0/lib/net/http/persistent.rb:692:in
start'
2019-08-13 21:48:37 +0000 [warn]: #0 /usr/local/bundle/gems/net-http-persistent-3.0.0/lib/net/http/persistent.rb:622:in connection_for'
2019-08-13 21:48:37 +0000 [warn]: #0 /usr/local/bundle/gems/net-http-persistent-3.0.0/lib/net/http/persistent.rb:927:in
request'
2019-08-13 21:48:37 +0000 [warn]: #0 /usr/local/bundle/gems/fluent-plugin-splunk-hec-1.1.0/lib/fluent/plugin/out_splunk_hec.rb:355:in send_to_hec'
2019-08-13 21:48:37 +0000 [warn]: #0 /usr/local/bundle/gems/fluent-plugin-splunk-hec-1.1.0/lib/fluent/plugin/out_splunk_hec.rb:167:in
write'
2019-08-13 21:48:37 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.4.0/lib/fluent/plugin/output.rb:1125:in try_flush'
2019-08-13 21:48:37 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.4.0/lib/fluent/plugin/output.rb:1425:in
flush_thread_run'
2019-08-13 21:48:37 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.4.0/lib/fluent/plugin/output.rb:454:in block (2 levels) in start'
2019-08-13 21:48:37 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.4.0/lib/fluent/plugin_helper/thread.rb:78:in
block in thread_create'
2019-08-13 21:48:37 +0000 [trace]: #0 enqueueing all chunks in buffer instance=69875636192300

2019-08-13 22:02:44 +0000 [error]: #0 failed to flush the buffer, and hit limit for retries. dropping all chunks in the buffer queue. retry_times=3 records=2 error_class=Net::OpenTimeout error="execution expired"
2019-08-13 22:02:44 +0000 [error]: #0 suppressed same stacktrace

Nevertheless, I can successfully send the same request from the node using Curl:

curl -k https://138.85.114.182:8088/services/collector/event -H "Authorization: Splunk ff6e61af-b78f-4e4a-a8ff-65d155a8c9b8" -d '{"host":"","time":"1565732853.4252117","index":"vecc_csa_kubernetes_logs","source":"namespace:splunk/pod:splunk-connect-splunk-kubernetes-objects-6cb5876855-rmqpf","sourcetype":"kube:objects:pods","fields":{"cluster_name":"cluster_name"},"event":"{\"metadata\":{\"name\":\"splunk-connect-splunk-kubernetes-objects-6cb5876855-rmqpf\",\"generateName\":\"splunk-connect-splunk-kubernetes-objects-6cb5876855-\",\"namespace\":\"splunk\",\"selfLink\":\"/api/v1/namespaces/splunk/pods/splunk-connect-splunk-kubernetes-objects-6cb5876855-rmqpf\",\"uid\":\"f3dc7102-be13-11e9-8dde-fa163e5b4bad\",\"resourceVersion\":\"5401185\",\"creationTimestamp\":\"2019-08-13T21:47:31Z\",\"labels\":{\"app\":\"splunk-kubernetes-objects\",\"engine\":\"fluentd\",\"pod-template-hash\":\"6cb5876855\",\"release\":\"splunk-connect\"},\"annotations\":{\"checksum/config\":\"b14e319dbcdaddd52e542e9fe9b529381b0f918591820bcdfc4a10fd2b572fff\"},\"ownerReferences\":[{\"apiVersion\":\"apps/v1\",\"kind\":\"ReplicaSet\",\"name\":\"splunk-connect-splunk-kubernetes-objects-6cb5876855\",\"uid\":\"f3da97df-be13-11e9-8dde-fa163e5b4bad\",\"controller\":true,\"blockOwnerDeletion\":true}],\"requestResourceVersion\":\"5401187\"},\"spec\":{\"volumes\":[{\"name\":\"conf-configmap\",\"configMap\":{\"name\":\"splunk-connect-splunk-kubernetes-objects\",\"defaultMode\":420}},{\"name\":\"secrets\",\"secret\":{\"secretName\":\"splunk-kubernetes-objects\",\"defaultMode\":420}},{\"name\":\"splunk-connect-splunk-kubernetes-objects-token-4m9w6\",\"secret\":{\"secretName\":\"splunk-connect-splunk-kubernetes-objects-token-4m9w6\",\"defaultMode\":420}}],\"containers\":[{\"name\":\"splunk-fluentd-k8s-objects\",\"image\":\"k8s-registry.eccd.local/splunk/kube-objects:1.1.0\",\"args\":[\"-c\",\"/fluentd/etc/fluent.conf\"],\"env\":[{\"name\":\"MY_NAMESPACE\",\"valueFrom\":{\"fieldRef\":{\"apiVersion\":\"v1\",\"fieldPath\":\"metadata.namespace\"}}},{\"name\":\"MY_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"apiVersion\":\"v1\",\"fieldPath\":\"metadata.name\"}}},{\"name\":\"NODE_NAME\",\"valueFrom\":{\"fieldRef\":{\"apiVersion\":\"v1\",\"fieldPath\":\"spec.nodeName\"}}},{\"name\":\"SPLUNK_HEC_TOKEN\",\"valueFrom\":{\"secretKeyRef\":{\"name\":\"splunk-kubernetes-objects\",\"key\":\"splunk_hec_token\"}}}],\"resources\":{\"requests\":{\"cpu\":\"100m\",\"memory\":\"200Mi\"}},\"volumeMounts\":[{\"name\":\"conf-configmap\",\"mountPath\":\"/fluentd/etc\"},{\"name\":\"secrets\",\"readOnly\":true,\"mountPath\":\"/fluentd/etc/splunk\"},{\"name\":\"splunk-connect-splunk-kubernetes-objects-token-4m9w6\",\"readOnly\":true,\"mountPath\":\"/var/run/secrets/kubernetes.io/serviceaccount\"}],\"terminationMessagePath\":\"/dev/termination-log\",\"terminationMessagePolicy\":\"File\",\"imagePullPolicy\":\"IfNotPresent\"}],\"restartPolicy\":\"Always\",\"terminationGracePeriodSeconds\":30,\"dnsPolicy\":\"ClusterFirst\",\"serviceAccountName\":\"splunk-connect-splunk-kubernetes-objects\",\"serviceAccount\":\"splunk-connect-splunk-kubernetes-objects\",\"nodeName\":\"worker-pool1-xmbq8dza-csa-eccd-v2\",\"securityContext\":{},\"schedulerName\":\"default-scheduler\",\"tolerations\":[{\"key\":\"node.kubernetes.io/not-ready\",\"operator\":\"Exists\",\"effect\":\"NoExecute\",\"tolerationSeconds\":300},{\"key\":\"node.kubernetes.io/unreachable\",\"operator\":\"Exists\",\"effect\":\"NoExecute\",\"tolerationSeconds\":300}],\"priority\":0,\"enableServiceLinks\":true},\"status\":{\"phase\":\"Running\",\"conditions\":[{\"type\":\"Initialized\",\"status\":\"True\",\"lastProbeTime\":null,\"lastTransitionTime\":\"2019-08-13T21:47:31Z\"},{\"type\":\"Ready\",\"status\":\"True\",\"lastProbeTime\":null,\"lastTransitionTime\":\"2019-08-13T21:47:32Z\"},{\"type\":\"ContainersReady\",\"status\":\"True\",\"lastProbeTime\":null,\"lastTransitionTime\":\"2019-08-13T21:47:32Z\"},{\"type\":\"PodScheduled\",\"status\":\"True\",\"lastProbeTime\":null,\"lastTransitionTime\":\"2019-08-13T21:47:31Z\"}],\"hostIP\":\"10.0.10.2\",\"podIP\":\"192.168.20.23\",\"startTime\":\"2019-08-13T21:47:31Z\",\"containerStatuses\":[{\"name\":\"splunk-fluentd-k8s-objects\",\"state\":{\"running\":{\"startedAt\":\"2019-08-13T21:47:32Z\"}},\"lastState\":{},\"ready\":true,\"restartCount\":0,\"image\":\"k8s-registry.eccd.local/splunk/kube-objects:1.1.0\",\"imageID\":\"docker-pullable://k8s-registry.eccd.local/splunk/kube-objects@sha256:209ea3157615eec0bd43bcae5450c4a7e2c962ce5c8dd42e890ca81dfc551103\",\"containerID\":\"docker://4c59e1927a3d386d8707e86033c73e9a82a7e56823811572abefffdaceda5e79\"}],\"qosClass\":\"Burstable\"}}"}'

{"text":"Success","code":0}

HEC is configured with insecureSSL=true.
Also, the same HEC configuration is being used with SCK for sending logging and metrics to Splunk. The logging and metrics agents are working fine.

Do you have any idea what could be wrong?

0 Karma

morawat
New Member

I fixed the issue by decreasing the "chunk_limit_size".

0 Karma
Get Updates on the Splunk Community!

Splunk Answers Content Calendar, June Edition

Get ready for this week’s post dedicated to Splunk Dashboards! We're celebrating the power of community by ...

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

See your relevant APM services, dashboards, and alerts in one place with the updated ...

As a Splunk Observability user, you have a lot of data you have to manage, prioritize, and troubleshoot on a ...