We have a notification service that has a series of four services, a web API, a fanout service that converts submitted multiple-recipient, multiple-delivery-method notifications into multiple notifications with just one recipient and one delivery method, and then delivery and retry services. Based on logging to Splunk as each notification is processed by each service (so states of "submitted" "fanned out", "delivered" and "pending retry"). The log events would have an ID associated with the notification, and the state that just completed.
I am hoping to identify notifications that are missing states, like "submitted" appears as a logged event, but no others, or "submitted" and "fanned out", appear, but nothing else. Notifications expire, so bonus points if anyone can come up with a way to track "submitted", "fanned out", "pending retry", but stopped getting "pending retry" log events before the notification expired. "delivered" is of course the final state.
Another way to think about this is looking for any "submitted" notification ID that does not have at least "fanned out" and "delivered".
I'm willing to set aside the complexity of the one-to-many relationship for now, unless someone has idea(s) about that. In other words, if the submitted notification has 3 recipients and 2 delivery methods, that should become 6 notifications. I'd love to be able to track that properly too, and I could log additional data to facilitate it if needed.
