We are on Splunk 7.3.1 , would need to know if there is any to de-duplicate the incoming data on FOrwarder or Indexer level?

[rashi83]

We have many instances where duplicate data has been coming due to server instances running in wrong manner. Because of that duplicate has found to be indexed.
Is there is config change we can do on Forwarder or Indexer to stop indexing supplicate entries.

[richgalloway · Accepted solution]

Neither forwarder nor indexer can deduplicate events at index time. You must correct the manner in which the server instances run.
You can, however, use dedup to deduplicate events at search time or use the delete command to hide the duplicates.