Why am I Getting Duplicated Data Using HEC Ingestion Method

[wsmworkhard]

I'm getting duplicated data when using lambda function to send events from cloudwatch to splunk through HTTP Event Collector. I didn't enable the indexer acknowledgement . Does anyone have the same issue before?

Regards,
Simeng

[splunkoptimus]

I'm having the same problem did you manage to fix it?

[PickleRick]

1. This is a very old thread. People involved probably don't use this forum anymore. Instead of hijacking ancient threads it's better to create a new one describing the problem and debugging steps alreadh taken. Possibly linking to the old thread for refetence.

2. The issue is so vaguely specified it's difficult to advise anything. We don't know what and how you're sending, we don't know how your receiving side is configured. We don't know the extent of your problem - are all your events duplicated or just some od them? We don't know if you did any debugging at all.

[jbuxton]

Well it's only been 2 years...

Did you get any help with this? I'm looking at cloudwatch logs that have 43 duplicated records in some cases.

While I'm planning to follow up with the team that implemented the lambda, I did discover that a implementation from another team is doing the same thing.

Indexer acknowledgment is enabled, but we are using Splunk Cloud - so it's either not supported, or limited to use with the Kinesis Firehose (depending on which documentation you read). Furthermore, I don't think any of the teams has implemented a 'dead letter queue'.

Any feedback appreciated - this is bad!