Is it possible to pass event id to Http Event Collector (HEC) to avoid indexing if same event is sent twice?

[knst]

My app notifies Splunk with the call to HEC on data changes. As data actually stored as series of events, it is quite straightforward to use Splunk for analysis. But, due to some internal reasons, it is possible that same events will be delivered to HEC twice. And it is crucial to have only one event stored in Splunk in this case.

Most obvious way to achieve this is to have some unique id posted with event and having Splunk ignore the event if it has id matching any of previously indexed events. But, I failed to find anything like this in the documentation.

[FrankVl]

I'm not aware of any way to make Splunk detect such duplicates at index time. Events are processed one by one, without knowledge of what was already indexed in the past. Why exactly is it crucial to only store 1 event in Splunk?

You can of course write your searches such that they deduplicate the data before presenting it on a dashboard / report / whatever.

[knst]

Why exactly is it crucial to only store 1 event in Splunk?
I was planning to use splank for data analysis: app publishes events for everything that is interesting from business point of view and splunk shows dashboards and reports to monitor the whole picture. But it looks like splunk does not fit my needs - e.g. if dashboard present number of events of given type or accumulates data from some events based on the type the data would be wrong if the same event is processed twice.

[FrankVl]

Well, you can write your Splunk dashboards and reports just the way you want them. And if you are able to let your application include unique IDs in the events, that enables you to detect duplicates, it is very easy to let splunk run a dedup command on the data, as part of the search time processing. That way, the duplicates are still in Splunk, but can be removed before it is presented on a dashboard/report or taken into account for some statistics or other calculations.

[knst]

makes sense... But as storage space required increases significantly with such approach.
Is possible to clean duplicates somehow? E.g. run some script on nightly basis...

[sandeeprachuri]

@knst, This approach uses more Splunk license and storage space. When it comes to clean, Either use Delete keyword or Clean CLI command. However, Delete only hides from visibility and you won't get your storage back. Clean can be used to flush out data from storage. Use it with very caution. But, I have never tried to clean part of an index [however, cleaned entire index].

Best thing is to filter duplicates before sending to Splunk.

PS: Splunk doesn't suggest clean on PRODuction systems.

Thanks,
Sandeep

[starcher]

That is not something HEC does. HEC is meant for developers to send in what they want so the burden is on the sending client to only send what it should.

[knst]

Agree - the question is rather about splunk itself than about HEC that is actually just a transport.
But the app that sends events uses event sourcing and thus eventual consistency is something I need to deal with and guaranty idempotency... I have full control on what is sent to splunk in terms of format - fields, metadata etc but there is no guarantee that same event would not delivered twice.