This data occurs in real time, and I receive it with bundles.
Each source send about 1000~2000 data for average in a day.
I want to compare _indextime and _time(_time is same as the time at the event log's has)
I used query like this.
| eval indextime=_indextime
| stats values(source) by indextime _time
| eval time_gap=indextime - _time, indextime=strftime(indextime, "%y/%m/%d %H:%M:%S")
I guessed the indextime will be always later than _time, because all events will indexing after the events get over.
Most of results are came out as I gussed, but some events was wiered.
indextime _time Time_gap
2018-03-22 1:05 2018-03-22 23:19 -1334
2018-03-22 1:05 2018-03-22 23:25 -1340
2018-03-22 1:05 2018-03-22 23:34 -1349
2018-03-22 1:05 2018-03-22 23:56 -1371
Why some indextime is later than _time?
try to see the latency,
* | eval time=_time | eval itime=_indextime | eval latency=(itime - time) | stats count, avg(latency), min(latency), max(latency) by source
Negative latencies, if observed, usually indicate a system clock difference between the source server (log-source) and the Splunk-server(indexer).
This figure will of course be inaccurate when Splunk indexes historic logs.- just to let you know