I'm pretty someone has asked the question before but couldn't find the post.
I'm trying to find a good way to search for events that have never happened before (for alerting purposes).
I've got two options.
Option 1 is based on using the punct field. I'll do something like rare punct | where punct <x and i'll be able to see the logs that have "weird" patterns. Problem is, if a log has the same format than another one, i won't be able to detect it as "never happened before".
Option 2 is based on a succession of "NOT" that would exclude everything i know from the results. This search takes a while to settle but that's fine, i've got time 😉
In both cases, when my search will generate a result i'll either :
- update the search so that the event does not generate another alert in the future (if it's considered "safe")
- leave the search this way as i'm interested in detected this kind of events
As anyone got a better solution ?
Little old but ran across this while searching for something else. Here is how I would attempt to tackle this.
Obviously a number of complexities based on your environment but I would look at a two stage approach. The first is create a scheduled search that outputs a csv containing known events. The second stage is to create your alert (again scheduled) with a subsearch that builds your NOT statement. Take Windows events for example. In my first scheduled search I might build my csv that contains EventCode, host, date first seen, sum of events since first seen, and most recent time of the EventCode & host pair. Schedule that to run once or twice a day (or whatever). Let that bake for a scheduled run or two.
The second stage is to create your alert. To extend the Windows example you might create your query like
index=whatever sourcetype=wineventlog* NOT [ | inputlookup known_events.csv | fields host EventCode] | ...
The 'trick' is to figure out what sort of thresholds your are comfortable with. The primary driver to me is how often you want the second search to run. In other words if only want that to run once a day then you probably only need the first search to run once per day as well (scheduled to finish before the second starts). If you wanted the second search to run once every hour you could still have the first search run once a day but there is a potential of getting 23 alerts (ie., a new system starts sending data).
I'm pretty sure that there's no very fast AND easy way of doing this. There are a number of search commands that you could use for coming closer to a solution that fits you, these include
rare. The issue you will likely run into with these commands (and any other solution I can think of) is that you would have to search over all time in order to have Splunk "remember" any event that has happened before. If that's OK with you, I definitely recommend looking up all these commands and see if anyone fits your needs.
On a sidenote, I know of a pretty cool solution that does exactly this (finds "unexpected / never happened before" events in a more or less infinite time window), but it's yet to be released.
Will try anomalies and anomalousvalue when i have a few minutes. Thanks for the tip !
What about this cool solution you are talking about ? I'm currently testing splunk 5 beta. Haven't seen such a thing though ... Can you tell me more about it ?
Hi there, that's not really what i'm trying to do here. I need splunk to "remember" events that might have happened à couple of years ago. So i can't really afford to use alert throttling 😕
It's not really "never happened before" but have you considered alert throttling? I'm not sure how long time period you can define (and will it kill the performance) but in theory something like this* could do almost what you want.