Dashboards & Visualizations

How to identify items that stopped being processed?

New Member

We have a notification service that has a series of four services, a web API, a fanout service that converts submitted multiple-recipient, multiple-delivery-method notifications into multiple notifications with just one recipient and one delivery method, and then delivery and retry services. Based on logging to Splunk as each notification is processed by each service (so states of "submitted" "fanned out", "delivered" and "pending retry"). The log events would have an ID associated with the notification, and the state that just completed.

I am hoping to identify notifications that are missing states, like "submitted" appears as a logged event, but no others, or "submitted" and "fanned out", appear, but nothing else. Notifications expire, so bonus points if anyone can come up with a way to track "submitted", "fanned out", "pending retry", but stopped getting "pending retry" log events before the notification expired. "delivered" is of course the final state.

Another way to think about this is looking for any "submitted" notification ID that does not have at least "fanned out" and "delivered".

I'm willing to set aside the complexity of the one-to-many relationship for now, unless someone has idea(s) about that. In other words, if the submitted notification has 3 recipients and 2 delivery methods, that should become 6 notifications. I'd love to be able to track that properly too, and I could log additional data to facilitate it if needed.

Labels (2)
0 Karma
Get Updates on the Splunk Community!

Build Scalable Security While Moving to Cloud - Guide From Clayton Homes

 Clayton Homes faced the increased challenge of strengthening their security posture as they went through ...

Mission Control | Explore the latest release of Splunk Mission Control (2.3)

We’re happy to announce the release of Mission Control 2.3 which includes several new and exciting features ...

Cloud Platform | Migrating your Splunk Cloud deployment to Python 3.7

Python 2.7, the last release of Python 2, reached End of Life back on January 1, 2020. As part of our larger ...