Recently a real-time search and email alert has failed to fire consistently. I am fairly certain that this used to work for every event. Now for some real-time triggered events, no email is getting sent. I compare the splunk search to the emails I'm getting and I am definitely missing emails. The search string has not changed. Perhaps it's a performance issue? (I am using a JOIN but my splunk admin tells me the system has plenty of resources). I am mostly curious about how to troubleshoot something like this. Thank you.
Update: I created the same real time alert without using a join (however I do need a join) and I am getting more emails for that alert than the one with the join. Could this be a timing or resource thing?