You may be running into timeout issues of various kinds, as well as, potentially, out-of-memory issues.
It would seem to be a more scalable approach, although a PITA to redo all the architecture, to send those all to a summary index as they are generated.
Since that would require a bunch of mantime, just check whether your above strategy would work correctly for, say, the last 15 minutes. If it does, then run it every 15 minutes to send the results to a summary index, and you can scan the summary index to see what happened over the last day.
... View more