I'm pretty someone has asked the question before but couldn't find the post.
I'm trying to find a good way to search for events that have never happened before (for alerting purposes).
I've got two options.
Option 1 is based on using the punct field. I'll do something like rare punct | where punct <x and i'll be able to see the logs that have "weird" patterns. Problem is, if a log has the same format than another one, i won't be able to detect it as "never happened before".
Option 2 is based on a succession of "NOT" that would exclude everything i know from the results. This search takes a while to settle but that's fine, i've got time 😉
In both cases, when my search will generate a result i'll either :
- update the search so that the event does not generate another alert in the future (if it's considered "safe")
- leave the search this way as i'm interested in detected this kind of events
As anyone got a better solution ?
It's not really "never happened before" but have you considered alert throttling? I'm not sure how long time period you can define (and will it kill the performance) but in theory something like this* could do almost what you want.
Hi there, that's not really what i'm trying to do here. I need splunk to "remember" events that might have happened à couple of years ago. So i can't really afford to use alert throttling 😕
I'm pretty sure that there's no very fast AND easy way of doing this. There are a number of search commands that you could use for coming closer to a solution that fits you, these include
rare. The issue you will likely run into with these commands (and any other solution I can think of) is that you would have to search over all time in order to have Splunk "remember" any event that has happened before. If that's OK with you, I definitely recommend looking up all these commands and see if anyone fits your needs.
On a sidenote, I know of a pretty cool solution that does exactly this (finds "unexpected / never happened before" events in a more or less infinite time window), but it's yet to be released.
Will try anomalies and anomalousvalue when i have a few minutes. Thanks for the tip !
What about this cool solution you are talking about ? I'm currently testing splunk 5 beta. Haven't seen such a thing though ... Can you tell me more about it ?
Please let me know when you're free to talk. I'm very interested and could not confirm the info with my contacts at Splunk.
Thanks a lot in advance !