In Splunk 7.0.0, when sending data to a metrics index, it looks like one can send duplicate metric measurement events (e.g., the same tuple of time, metric name, and dimensions) and the metric index ...
See more...
In Splunk 7.0.0, when sending data to a metrics index, it looks like one can send duplicate metric measurement events (e.g., the same tuple of time, metric name, and dimensions) and the metric index will store all duplicates, thereby affecting the statistics that come out.
Is that the intended behavior for the metric index?
Other time-series metric stores/indexes/dbs I have played with use overwrite/last-in logic that only preserves the most-recently indexed value for a given metric tuple. Using similar logic here would seem to make more sense for the the use cases I would see for the metric store, but I freely admit to making assumptions.
Please clarify how allowing duplicate metric events is intended to be used / handles.
Note, my understanding of a distinct metric tuple is the timestamp (to milliseconds), metric name, and dimension fields. So, assuming you see the following two metric tuples that arrive at the indexer at different times (the first column), only the later one (the top row) would be saved in the index. Right now (as of Splunk 7.0.0), both are saved in the metrics index/store.
| indexing timestamp | metric timestamp | metric name | metric value | server
| 1506708015.390 | 1506708000.000 | server.power.kw | 126.06 | na-server-1
| 1506708010.242 | 1506708000.000 | server.power.kw | 104.56 | na-server-1
Additional Comments after posting
The example data I provided above is simply made-up in order to simplify the discussion. Don't interpret it as relevant to the question - it's just an example.
Some points to consider:
At least two other time-series databases for metrics don't allow duplicate events: InfluxDB and OpenTSDB. Haven't fully evaluated others (e.g., DataDog), just using these as examples that I know of.
Splunk's documentation openly says individual events are not really relevant in metric indexes,
you cannot filter or search on the metric value field in mstats
delete command doesn't work for metrics ( i.e., you can't delete duplicates if they happen)
By allowing duplicate tuples {timestamp, metric name, dimensions} in metrics indexes, backfilling via saved searches or resending of metrics becomes very, very difficult. Backfilling using metrics distilled from event indexes is very easy if you use write-last / last-in / overwriting logic.
Running aggregate metric sources (like my example above - total power consumed in an hour) become very challenging with current, duplicate metric logic
Clustered environments raise the risk of getting duplicate events in the face of delays / blocked queues and resent events
So, maybe best to sum up as this question: is the Splunk metrics index feature intended to work like other time-series databases with strict write logic limitations, or is it an optimized / pared-down version of the standard Splunk event index?