Why are we seeing duplicate events found in an index after update to Splunk 6.3.x?

[hkaiser]

We use several scheduled reports to ensure that we do not have any duplicate events in our indexes.
Our searches look as follows:

index=ng  | eval myUniqueId = index + "_" + _cd + "_" + splunk_server | stats count by myUniqueId | where count > 1

AND

index=ng | streamstats count as DuplicateNumber by _raw | search DuplicateNumber>1

Above searches worked fine until splunk 6.2 and did not find any duplicate events for index ng.
We did upgrade Splunk in the last weeks to version 6.3.1 and last week also to version 6.3.3. For Splunk 6.3, the above searches do not return correct results.

If we run above searches for the last 7 days, then I get duplicates e.g. for 04.02.2016 between 00:00 and 02:00 UTC. However, if I limit the same search to search for duplicates on 04.02.2016 between 00:00 and 02:00 UTC, then no duplicates are returned - this doesn't make any sense.

It get's even more confusing: If we afterwards limit the time window of the search to 02.02.2016 00:00 to 05.02.2016 00:00 (which would include the duplicates, that have been detected at 04.02.2016 between 00:00 and 02:00 UTC) I get now only duplicates on 03.02.2016 between 00:00 and 02:00 - but no duplicates on the 04.02.2016. So it looks like there is something seriously broken.

Also if we explicitly search for an event, that according to above searches in Splunk 6.3 is a duplicate, we only find this event once.

After above results, we stopped Splunk 6.3 and moved the index to a different Splunk instance, that has still version 6.2: Above searches worked fine on Splunk 6.2 again - so we do not assume that the index ng is broken somehow.

Can anybody confirm our observations?
Thank you.

[jplumsdaine22]

When you say you ran the same query on 6.2, you have a 6.2 search head or this is a completely separate indexer?

When you run the search is the output you get?

| _raw |  _time |count | max(DuplicateNumber) |
| xxxx | 12:00  | 1    | 2                    | 

or are there 7267 rows?

[gtriSplunk]

Are you running a clustered index environment that is multi-site?

[hkaiser]

No clustered index. One host only that runs all splunk components (indexer, search head). Input comes from local files on this host.
This setup worked fine for more than 1 year. Problems started with the upgrade from 6.2 to 6.3.