Kinesis error when consuming from cloudwatch logs queue

[Jeremiah]

Has anyone had success with connecting Kinesis to Cloudwatch logs to pull data into Splunk? I've used the Kinesis modular input to pull other events, but when using the new Cloudwatch logs subscription feature, the Kinesis modular input returns an error for every event in the queue:

07-02-2015 11:13:26.029 -0700 ERROR ExecProcessor - message from "python /local/mnt/splunk/stage/splunk/etc/apps/kinesis_ta/bin/kinesis.py" Couldn't process record {SequenceNumber:     49552014282308283371136136273926985644062839320977866754,Data: java.nio.HeapByteBuffer[pos=1 lim=782 cap=782],PartitionKey: 035694369293:PiwikAccessLogs:awspiwikstg02.qualcomm.com}. Skipping the record.
07-02-2015 11:13:33.237 -0700 ERROR ExecProcessor - message from "python /local/mnt/splunk/stage/splunk/etc/apps/kinesis_ta/bin/kinesis.py" Malformed data: null
07-02-2015 11:13:33.238 -0700 ERROR ExecProcessor - message from "python /local/mnt/splunk/stage/splunk/etc/apps/kinesis_ta/bin/kinesis.py" java.nio.charset.MalformedInputException: Input length = 1

Using the AWS cli, we can consume these messages, and they are definitely not null. And if we push a message onto the same queue from the cli, that works and is consumed just fine in Splunk using the modular input.

[Damien_Dallimor]

Presumably it must be something to do with the way Cloudwatch writes data to Kinesis , that essentially means you have to use an actual Cloudwatch client (rather than trying to sneak in the backdoor and hit the underlying Kinesis queues directly)

[Jeremiah]

I agree it must be something to do with the way that Cloudwatch writes the data, but this is not hitting the underlying kinesis queues. This is using a new feature of CWL called subscriptions, that writes the CWL entries to a kinesis queue specifically so you can consume the logs with other applications.

[Jeremiah]

It looks like the reason for the error is that the CWL events in kinesis are encoded and compressed:

http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/Subscriptions.html

"The Data attribute in the Amazon Kinesis record is Base64 encoded and compressed with the gzip format"