Journey monitoring and alerting


We're currently using splunk for  traditional dashboards, monitoring and alerting and it means that we're now very effective at identifying and addressing errors and exceptions in our apps when they occur.

We're now looking to build more sophisticated monitoring that looks for issues across journeys that users complete in our app and helps us to identify more subtle issues that might not involve errors or exceptions.

For example a simplified version of our sign up process looks something like

1. Receive sign up request

2. Create account record

3. Write account created message to queue

4. Read account created from message queue and send welcome email

Each of the steps are logged in splunk and there's a common correlation id logged across each of the steps. Any exceptions that occur trigger alerts so that's all good.

On occasion we might have an issue where messages at step 4 stop being read from the queue and the welcome emails are not sent but nothings throwing exceptions and it's not obvious anythings wrong until a customer contacts us to flag the missing sign up email.

At that point we can query splunk and see that steps 1, 2, and 3 completed successfully, but there's no logs for step 4 which indicates an issue that need investigating.

We'd like to be able to automated the process for checking that all of the expected steps in a given journey are completed and alert when steps are missed. Is there a way we can achieve this with splunk?

I've seen a question around visualising order journeys ( which sounds in the same ballpark and refers to Splunk Business Flow, but the docs indicate it's no longer available to purchase... Are there other out of the box or paid options?

Labels (1)
0 Karma


You may not need anything fancy like Business Flow.  Perhaps all you need is a simple query that searches for all 4 events and correlates them by ID.  Throw an alert if the query sees a set of less than 4 events in an appropriate amount of time.

If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Five Subtly Different Ways of Adding Manual Instrumentation in Java

You can find the code of this example on GitHub here. Please feel free to star the repository to keep in ...

New Splunk APM Enhancements Help Troubleshoot Your MySQL and NoSQL Databases Faster

Splunk Observability has two new enhancements to make it quicker and easier to troubleshoot slow or frequently ...

How to Troubleshoot our Splunk HEC Endpoint

This blog post is part of an ongoing series on OpenTelemetry. In this blog post, we will explore the best way ...