All Apps and Add-ons

Multiple Kinesis inputs - GetShardIterator errors

Explorer

We have created Kinesis streams in multiple regions within the same account. Each stream has the same name, though a different arn due to the service being region specific (e.g. arn:aws:kinesis:us-west-2:123456789012:stream/), so they are distinct and distinguishable. The logs are showing errors below for all but one of the configured streams.

"An error occurred (InvalidArgumentException) when calling the GetShardIterator operation: StartingSequenceNumber 49575723201104542708550335494573616233923858177688862722 used in GetShardIterator on shard shardId-000000000000 in stream under account 870296345612 is invalid because it did not come from this stream."

It looks like this might be a bug in the way Kinesis data is getting loaded, but perhaps there's a setting in the conf that needs to be added?

0 Karma

Explorer

We're seeing this same error now with Kinesis inputs. Was there any resolution?

0 Karma

Explorer

We talked with Splunk directly on this (and some other issues) though never got any follow up. We ended up just re-naming the streams to append region name to the end and the data started coming in as expected afterward. Not a real 'resolution', but it was a workaround that fixed the behavior.

Explorer

Thanks. We actually just got confirmation from Splunk that each Kinesis stream name must be unique for each account and region. This was not documented.

The app really should check for this during input setup...

Ultra Champion

You've tagged 2 entirely different apps in your question.Which app are you using ?

0 Karma

Explorer

Apologies, I didn't realize the Kinesis Modular Input was separate from the input type found in the Splunk App for AWS. The app in use is the Splunk App for AWS.

0 Karma

Ultra Champion

Ok , I wrote the other one that is specific to pulling in binary/text data from Kinesis 🙂

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!