Multiple Kinesis inputs - GetShardIterator errors

[rjhenry3]

We have created Kinesis streams in multiple regions within the same account. Each stream has the same name, though a different arn due to the service being region specific (e.g. arn:aws:kinesis:us-west-2:123456789012:stream/), so they are distinct and distinguishable. The logs are showing errors below for all but one of the configured streams.

"An error occurred (InvalidArgumentException) when calling the GetShardIterator operation: StartingSequenceNumber 49575723201104542708550335494573616233923858177688862722 used in GetShardIterator on shard shardId-000000000000 in stream under account 870296345612 is invalid because it did not come from this stream."

It looks like this might be a bug in the way Kinesis data is getting loaded, but perhaps there's a setting in the conf that needs to be added?

[rvasaly]

We're seeing this same error now with Kinesis inputs. Was there any resolution?

[rjhenry3]

We talked with Splunk directly on this (and some other issues) though never got any follow up. We ended up just re-naming the streams to append region name to the end and the data started coming in as expected afterward. Not a real 'resolution', but it was a workaround that fixed the behavior.

[rvasaly]

Thanks. We actually just got confirmation from Splunk that each Kinesis stream name must be unique for each account and region. This was not documented.

The app really should check for this during input setup...

[Damien_Dallimor]

You've tagged 2 entirely different apps in your question.Which app are you using ?

[rjhenry3]

Apologies, I didn't realize the Kinesis Modular Input was separate from the input type found in the Splunk App for AWS. The app in use is the Splunk App for AWS.

[Damien_Dallimor]

Ok , I wrote the other one that is specific to pulling in binary/text data from Kinesis 🙂