Splunk Connect for Kubernetes

[alexlombardi]

I'm currently setting up logging from an EKS cluster into Splunk and am using the Splunk Connect for Kubernetes. We've deployed this tool using the provided helm chart and followed the values.yaml to basically log everything that it can just so we can test, but I'm running into an issue where some logs are not getting sent to Splunk and I cant seem to figure out why. The basic EKS logs from various components are being picked up including an application log which shows events like the below, however only the first event/log is being picked up by Splunk (These logs are obfuscated):

{"log":" {\"MessageId\":\"80facc67-beef-1234-4567-1234b0afddcc\",\"ReceiptHandle\":\"aBc2d6Y2c1234/gZ2Tb1234/M6Tyzdf1234/76z1234gBh1234/zGb1234/M6ThB1234/Bghp12TcDqZ1234+/GBKsqxgcFWExq+GSYA=\",\"MD5OfBody\":\"049dfb3e741234\",\"Body\":\"{\\\"Records\\\":[{\\\"eventVersion\\\":\\\"2.1\\\",\\\"eventSource\\\":\\\"aws:s3\\\",\\\"awsRegion\\\":\\\"us-east-1\\\",\\\"eventTime\\\":\\\"2021-01-20T15:57:44.247Z\\\",\\\"eventName\\\":\\\"ObjectCreated:Put\\\",\\\"userIdentity\\\":{\\\"principalId\\\":\\\"AWS:BDG1234\\\"},\\\"requestParameters\\\":{\\\"sourceIPAddress\\\":\\\"1.1.1.1\\\"},\\\"responseElements\\\":{\\\"x-amz-request-id\\\":\\\"1GT6YWQERVC\\\",\\\"x-amz-id-2\\\":\\\"KGBtzde1fg6Fdz1234/uOgVxBtBdz1234\\\"},\\\"s3\\\":{\\\"s3SchemaVersion\\\":\\\"1.0\\\",\\\"configurationId\\\":\\\"test-name-here\\\",\\\"bucket\\\":{\\\"name\\\":\\\"test-name-here\\\",\\\"ownerIdentity\\\":{\\\"principalId\\\":\\\"GBT1RTYZ1234\\\"},\\\"arn\\\":\\\"arn:aws:s3:::test-name-here\\\"},\\\"object\\\":{\\\"key\\\":\\\"upload/test-name-here.xml\\\",\\\"size\\\":757,\\\"eTag\\\":\\\"6ts3Szdfa45hVx\\\",\\\"sequencer\\\":\\\"1234GBG1234\\\"}}}]}\",\"Attributes\":{\"SentTimestamp\":\"1611158266018\"}}\n","stream":"stdout","time":"2021-01-20T15:57:50.649004404Z"}

{"log":"[2021-01-20T15:57:50.648Z] INFO (LOGS/1): received message, new data: test-name-here upload/test-name-here.xml\n","stream":"stdout","time":"2021-01-20T15:57:50.649112304Z"}

The second log is never getting to Splunk and both of these events are in the same log file.

Looking over the logs from the HEC on my receivers I don't see any parsing errors or anything that would indicate why these second group of logs are not being picked up. I've also checked on the containers themselves and found no errors from fluentd to help explain the problem.

Has anyone else run into an issue like this and have  a solution to why some events would be parsed and sent properly and others wouldn't even though there in the same file? I've found this article about a similar issue, but no resolution was provided.

https://community.splunk.com/t5/All-Apps-and-Add-ons/Missing-logs-Splunk-Connect-for-Kubernetes/m-p/...