Getting Data In

Why am I Getting Duplicated Data Using HEC Ingestion Method

wsmworkhard
Explorer

I'm getting duplicated data when using lambda function to send events from cloudwatch to splunk through HTTP Event Collector. I didn't enable the indexer acknowledgement . Does anyone have the same issue before?

Regards,
Simeng

Tags (1)
0 Karma

splunkoptimus
Path Finder

I'm having the same problem did you manage to fix it?

0 Karma

PickleRick
SplunkTrust
SplunkTrust

1. This is a very old thread. People involved probably don't use this forum anymore. Instead of hijacking ancient threads it's better to create a new one describing the problem and debugging steps alreadh taken. Possibly linking to the old thread for refetence.

2. The issue is so vaguely specified it's difficult to advise anything. We don't know what and how you're sending, we don't know how your receiving side is configured. We don't know the extent of your problem - are all your events duplicated or just some od them? We don't know if you did any debugging at all.

jbuxton
Explorer

Well it's only been 2 years...

Did you get any help with this? I'm looking at cloudwatch logs that have 43 duplicated records in some cases.

While I'm planning to follow up with the team that implemented the lambda, I did discover that a implementation from another team is doing the same thing.

Indexer acknowledgment is enabled, but we are using Splunk Cloud - so it's either not supported, or limited to use with the Kinesis Firehose (depending on which documentation you read). Furthermore, I don't think any of the teams has implemented a 'dead letter queue'.

Any feedback appreciated - this is bad!

0 Karma
Get Updates on the Splunk Community!

AppDynamics Summer Webinars

This summer, our mighty AppDynamics team is cooking up some delicious content on YouTube Live to satiate your ...

SOCin’ it to you at Splunk University

Splunk University is expanding its instructor-led learning portfolio with dedicated Security tracks at .conf25 ...

Credit Card Data Protection & PCI Compliance with Splunk Edge Processor

Organizations handling credit card transactions know that PCI DSS compliance is both critical and complex. The ...