If I understand you correctly, you'd like to deduplicate events on ingest - either not ingest an event if there is already one with the same value of a field called ID or overwrite previous values of such field. Well, that's not possible with native splunk functionalities. 1. Splunk ingestion process works one event at a time. 2. Splunk ingestion process works "one-way" - you can't "check what's already in the index". Remember that parsing can be performed way, way before the event even reaches the indexers (and different events from the same source can be processed on different components). Also, you don't have access to search-time extracted values during the ingestion process. 3. There is no "overwriting" in Splunk. So if it's really essential for you that you don't ingest duplicated ID's, you need to design your own ingestion process that will keep your events deduplicated (but for that you'd need some buffer window which will increase latency).
... View more