I guess my problem is getting the correct information when events haven't happened yet.
So, how do I know what to trigger on when an event has not happened yet?
If it's a common application (such as say windows login) you can google for the exception. Or, if it's an homegrown app, ask your developers.
If there's a field in your data that represents auth status (success, etc.), you can set up your search using the != operator. (not equal to)
Or, if you know what the value WOULD be for a failed login, it's better to be specific and use
myAuthStatusField=failed (or denied, or whatever the value would potentially be)
Then set your search schedule to run every 5 minutes for a time range of the last 5 minutes. Set your alert conditions to "if number of events is greater than 9".
If you don't have that field defined and haven't extracted a new field before, you can read the full documentation here: http://docs.splunk.com/Documentation/Splunk/4.2.3/User/ExtractNewFields