We're currently using splunk for traditional dashboards, monitoring and alerting and it means that we're now very effective at identifying and addressing errors and exceptions in our apps when they occur.
We're now looking to build more sophisticated monitoring that looks for issues across journeys that users complete in our app and helps us to identify more subtle issues that might not involve errors or exceptions.
For example a simplified version of our sign up process looks something like
1. Receive sign up request
2. Create account record
3. Write account created message to queue
4. Read account created from message queue and send welcome email
Each of the steps are logged in splunk and there's a common correlation id logged across each of the steps. Any exceptions that occur trigger alerts so that's all good.
On occasion we might have an issue where messages at step 4 stop being read from the queue and the welcome emails are not sent but nothings throwing exceptions and it's not obvious anythings wrong until a customer contacts us to flag the missing sign up email.
At that point we can query splunk and see that steps 1, 2, and 3 completed successfully, but there's no logs for step 4 which indicates an issue that need investigating.
We'd like to be able to automated the process for checking that all of the expected steps in a given journey are completed and alert when steps are missed. Is there a way we can achieve this with splunk?
I've seen a question around visualising order journeys (https://community.splunk.com/t5/Dashboards-Visualizations/How-to-Visualize-Order-Journey-through-spl... which sounds in the same ballpark and refers to Splunk Business Flow, but the docs indicate it's no longer available to purchase... Are there other out of the box or paid options?
