This is my problem as well. I run a daily report to generate metrics from a larger index. (I need the raw information for some analytics, but only the metrics for others, and the metrics queries are faster this way). If the report is ever run twice (by error, or restart, or an external reason) that day's metrics are forever no longer valid, and no way obvious way to filter out the duplicates.
Because the metric is generated from 24 hours of input raw data, it can't be generated at intake time. As far as I know.
And to counter the answers above, if these metrics were external, and were input more than once, it would be the same problem.
In a perfect world, the duplicates should never occur, but it would be safer and superior if there was an option to update a metric instead of duplicate it.
... View more