We have an every minute summary indexing job which runs happily looking at data from a few minutes ago.
If we push a config out causing a captain change, we have seen twice in a row now that the new captain is looking at data 4 days ago. If we restart that instance and the original instance takes over, then it goes back to looking at the present time.
Last time this happened, we had to cycle through all three instances before we got back to the original. The second instance was running about 6 minutes behind, which, while not as bad, causes us duplicate information.
SH1 = currenty
SH2 = -6m
SH2 = -4days
The summary job does the right thing and attempts to backfill the data, but clearly this is not the behavior we are looking for.
What should I be looking for to start debugging this?