topic Re: Multiple Kinesis inputs - GetShardIterator errors in All Apps and Add-ons

Multiple Kinesis inputs - GetShardIterator errors

rjhenry3 — Wed, 13 Sep 2017 22:08:24 GMT

We have created Kinesis streams in multiple regions within the same account. Each stream has the same name, though a different arn due to the service being region specific (e.g. arn:aws:kinesis:us-west-2:123456789012:stream/), so they are distinct and distinguishable. The logs are showing errors below for all but one of the configured streams.

"An error occurred (InvalidArgumentException) when calling the GetShardIterator operation: StartingSequenceNumber 49575723201104542708550335494573616233923858177688862722 used in GetShardIterator on shard shardId-000000000000 in stream under account 870296345612 is invalid because it did not come from this stream."

It looks like this might be a bug in the way Kinesis data is getting loaded, but perhaps there's a setting in the conf that needs to be added?

Re: Multiple Kinesis inputs - GetShardIterator errors

Damien_Dallimor — Wed, 13 Sep 2017 22:35:26 GMT

You've tagged 2 entirely different apps in your question.Which app are you using ?

Re: Multiple Kinesis inputs - GetShardIterator errors

rjhenry3 — Thu, 14 Sep 2017 01:31:26 GMT

Apologies, I didn't realize the Kinesis Modular Input was separate from the input type found in the Splunk App for AWS. The app in use is the Splunk App for AWS.

Re: Multiple Kinesis inputs - GetShardIterator errors

Damien_Dallimor — Thu, 14 Sep 2017 03:17:43 GMT

Ok , I wrote the other one that is specific to pulling in binary/text data from Kinesis 🙂

Re: Multiple Kinesis inputs - GetShardIterator errors

rvasaly — Thu, 07 Dec 2017 19:47:01 GMT

We're seeing this same error now with Kinesis inputs. Was there any resolution?

Re: Multiple Kinesis inputs - GetShardIterator errors

rjhenry3 — Mon, 11 Dec 2017 19:01:06 GMT

We talked with Splunk directly on this (and some other issues) though never got any follow up. We ended up just re-naming the streams to append region name to the end and the data started coming in as expected afterward. Not a real 'resolution', but it was a workaround that fixed the behavior.

Re: Multiple Kinesis inputs - GetShardIterator errors

rvasaly — Mon, 11 Dec 2017 22:27:24 GMT

Thanks. We actually just got confirmation from Splunk that each Kinesis stream name must be unique for each account and region. This was not documented.

The app really should check for this during input setup...