All Apps and Add-ons

Why does Amazon Kinesis Modular Input use DynamoDB and throw this error?

carlkennedy
Path Finder

04-07-2016 22:12:23.892 -0400 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/kinesis_ta/bin/kinesis.py" Caused by: com.amazonaws.AmazonServiceException: User: arn:aws:iam::###:user/zzz is not authorized to perform: dynamodb:CreateTable on resource: arn:aws:dynamodb:us-east-1:226425452713:table/zzz (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: AccessDeniedException;

My goal is to just pull the stream into Splunk.

0 Karma
1 Solution

Jeremiah
Motivator

The modular input uses DynamoDB to track the last read message from the queue. The DynamoDB table name is set by the application value you enter when you setup the Kinesis stream in Splunk. You'll incur DynamDB charges for the table, but the charges should be minimal. The IAM account you are using to read from Kinesis should have rights to create the table (unless you manually create the table) and read/write access.

Also, regardless of your Kinesis stream location, the Kinesis modular input will create the table in us-east-1. I wanted to have the table and the stream in the same region. There are some minor edits you can make to the code to do this. Let me know if you need them and I'll share.

View solution in original post

0 Karma

Damien_Dallimor
Ultra Champion
0 Karma

mwiora
Explorer

@damien thanks!

0 Karma

Jeremiah
Motivator

The modular input uses DynamoDB to track the last read message from the queue. The DynamoDB table name is set by the application value you enter when you setup the Kinesis stream in Splunk. You'll incur DynamDB charges for the table, but the charges should be minimal. The IAM account you are using to read from Kinesis should have rights to create the table (unless you manually create the table) and read/write access.

Also, regardless of your Kinesis stream location, the Kinesis modular input will create the table in us-east-1. I wanted to have the table and the stream in the same region. There are some minor edits you can make to the code to do this. Let me know if you need them and I'll share.

View solution in original post

0 Karma

mwiora
Explorer

Thanks jeremiah,

actually this is a show-stopper. I paste the following lines to make this answer searchable for search machines:
I created an IAM user which had permissions to read from Kinesis Stream only and I'm getting:

02-07-2017 16:24:31.029 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kinesis_ta/bin/kinesis.py" at com.amazonaws.services.kinesis.leases.impl.LeaseManager.createLeaseTableIfNotExists(LeaseManager.java:124)
02-07-2017 16:24:31.029 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kinesis_ta/bin/kinesis.py" at com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibLeaseCoordinator.initialize(KinesisClientLibLeaseCoordinator.java:172)
02-07-2017 16:24:31.029 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kinesis_ta/bin/kinesis.py" at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.initialize(Worker.java:377)
02-07-2017 16:24:31.029 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kinesis_ta/bin/kinesis.py" at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.run(Worker.java:321)
02-07-2017 16:24:31.029 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kinesis_ta/bin/kinesis.py" at com.splunk.modinput.kinesis.KinesisModularInput$MessageReceiver.run(Unknown Source)
02-07-2017 16:24:31.029 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kinesis_ta/bin/kinesis.py" Caused by: com.amazonaws.AmazonServiceException: User: arn:aws:iam::xxxxxxx:user/splunk-fwd-ir is not authorized to perform: dynamodb:CreateTable on resource: arn:aws:dynamodb:us-east-1:xxxxxxx:table/splunk-fwd-ir (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: AccessDeniedException; Request ID: xxxxxxx)

I contact the developer of the application to ask if he could publish the source of the application.

Cheers,
µatthias

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!