I have an alert (something_failure) that checks every 5 minutes, and when it finds a problem, it posts a message to Slack with a "Show results in Splunk" link. The weird part: clicking that link doesn't behave the same way every time, even though it's the exact same alert every time. One time, the alert fired at 11:20 AM. I clicked the link at 11:32 AM (12 minutes later) and got: "The search you requested could not be found. The search has probably expired or been deleted." Another time, the same alert fired at 11:40 AM. I clicked the link at 12:14 PM (34 minutes later) and it worked completely fine. So sometimes the results disappear in under 12 minutes, and other times they're still there after 34+ minutes -- for the exact same alert, same schedule, same setup. My question is simply: why can't I always see the result, and why is it different each time? Could someone confirm: Is it normal for the same alert's results to sometimes last 10 minutes and sometimes last 30+ minutes? Or should it always be the same? Is there a way to make sure results stick around for a guaranteed amount of time, so this doesn't happen? Is there a better way to set up an alert so the Slack link always works, instead of sometimes dying early?
... View more