Why does Amazon Kinesis Modular Input use DynamoDB and throw this error?

[carlkennedy]

04-07-2016 22:12:23.892 -0400 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/kinesis_ta/bin/kinesis.py" Caused by: com.amazonaws.AmazonServiceException: User: arn:aws:iam::###:user/zzz is not authorized to perform: dynamodb:CreateTable on resource: arn:aws:dynamodb:us-east-1:226425452713:table/zzz (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: AccessDeniedException;

My goal is to just pull the stream into Splunk.

[Damien_Dallimor]

Source : https://github.com/damiendallimore/SplunkModularInputsJavaFramework/tree/master/kinesis

[mwiora]

@damien thanks!

[Jeremiah · Accepted solution]

The modular input uses DynamoDB to track the last read message from the queue. The DynamoDB table name is set by the application value you enter when you setup the Kinesis stream in Splunk. You'll incur DynamDB charges for the table, but the charges should be minimal. The IAM account you are using to read from Kinesis should have rights to create the table (unless you manually create the table) and read/write access.

Also, regardless of your Kinesis stream location, the Kinesis modular input will create the table in us-east-1. I wanted to have the table and the stream in the same region. There are some minor edits you can make to the code to do this. Let me know if you need them and I'll share.

[mwiora]

Thanks jeremiah,

actually this is a show-stopper. I paste the following lines to make this answer searchable for search machines:
I created an IAM user which had permissions to read from Kinesis Stream only and I'm getting:

02-07-2017 16:24:31.029 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kinesis_ta/bin/kinesis.py" at com.amazonaws.services.kinesis.leases.impl.LeaseManager.createLeaseTableIfNotExists(LeaseManager.java:124)
02-07-2017 16:24:31.029 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kinesis_ta/bin/kinesis.py" at com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibLeaseCoordinator.initialize(KinesisClientLibLeaseCoordinator.java:172)
02-07-2017 16:24:31.029 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kinesis_ta/bin/kinesis.py" at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.initialize(Worker.java:377)
02-07-2017 16:24:31.029 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kinesis_ta/bin/kinesis.py" at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.run(Worker.java:321)
02-07-2017 16:24:31.029 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kinesis_ta/bin/kinesis.py" at com.splunk.modinput.kinesis.KinesisModularInput$MessageReceiver.run(Unknown Source)
02-07-2017 16:24:31.029 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kinesis_ta/bin/kinesis.py" Caused by: com.amazonaws.AmazonServiceException: User: arn:aws:iam::xxxxxxx:user/splunk-fwd-ir is not authorized to perform: dynamodb:CreateTable on resource: arn:aws:dynamodb:us-east-1:xxxxxxx:table/splunk-fwd-ir (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: AccessDeniedException; Request ID: xxxxxxx)

I contact the developer of the application to ask if he could publish the source of the application.

Cheers,
µatthias