topic Re: Why am I Getting Duplicated Data Using HEC Ingestion Method in Getting Data In

Why am I Getting Duplicated Data Using HEC Ingestion Method

wsmworkhard — Tue, 07 May 2019 01:51:24 GMT

I'm getting duplicated data when using lambda function to send events from cloudwatch to splunk through HTTP Event Collector. I didn't enable the indexer acknowledgement . Does anyone have the same issue before?

Regards,
Simeng

Re: Why am I Getting Duplicated Data Using HEC Ingestion Method

jbuxton — Mon, 31 Jan 2022 23:08:21 GMT

Well it's only been 2 years...

Did you get any help with this? I'm looking at cloudwatch logs that have 43 duplicated records in some cases.

While I'm planning to follow up with the team that implemented the lambda, I did discover that a implementation from another team is doing the same thing.

Indexer acknowledgment is enabled, but we are using Splunk Cloud - so it's either not supported, or limited to use with the Kinesis Firehose (depending on which documentation you read). Furthermore, I don't think any of the teams has implemented a 'dead letter queue'.

Any feedback appreciated - this is bad!

Re: Why am I Getting Duplicated Data Using HEC Ingestion Method

splunkoptimus — Thu, 03 Jul 2025 18:45:31 GMT

I'm having the same problem did you manage to fix it?

Re: Why am I Getting Duplicated Data Using HEC Ingestion Method

PickleRick — Thu, 03 Jul 2025 19:40:14 GMT

1. This is a very old thread. People involved probably don't use this forum anymore. Instead of hijacking ancient threads it's better to create a new one describing the problem and debugging steps alreadh taken. Possibly linking to the old thread for refetence.

2. The issue is so vaguely specified it's difficult to advise anything. We don't know what and how you're sending, we don't know how your receiving side is configured. We don't know the extent of your problem - are all your events duplicated or just some od them? We don't know if you did any debugging at all.