Running the same search, why are different results showing up?

[szymonledzinski]

If I run the same search using the same time window I get sometimes different results.
I have added

| eval bkt=_bkt | stats count by splunk_server index bkt 

At the end of the search to check which buckets are being read.
For some reason splunk skips 1 or 2 buckets sometimes.
We are using indexer cluster (10 nodes, 2 search factor, 3 replication factor).
All Data is Searchable, Search Factor is Met and Replication Factor is Met.
I don't see any errors in search logs. Any ideas what could be a problem?

[nickhills]

Because buckets are uniquely named per indexer, it will depend which indexer in your cluster provides the results to your search.

Since you have a Search factor > 1, there are two or more copies of each bucket (which will have different names on each indexer since each indexer applies its GUID to the end)

Its not an exact science, but (if your data is well distributed) a search over a small time window should return ~10 buckets (ideally one from each indexer)
If you run that search later, it's conceivable that you could get 10 entirely different buckets returned (from different servers), whilst representing the exact same results. This is by design.