Splunk Search
Highlighted

Finding events that have never happened before

Communicator

Hello there,

I'm pretty someone has asked the question before but couldn't find the post.
I'm trying to find a good way to search for events that have never happened before (for alerting purposes).

I've got two options.
Option 1 is based on using the punct field. I'll do something like rare punct | where punct <x and i'll be able to see the logs that have "weird" patterns. Problem is, if a log has the same format than another one, i won't be able to detect it as "never happened before".
Option 2 is based on a succession of "NOT" that would exclude everything i know from the results. This search takes a while to settle but that's fine, i've got time 😉

In both cases, when my search will generate a result i'll either :
- update the search so that the event does not generate another alert in the future (if it's considered "safe")
- leave the search this way as i'm interested in detected this kind of events

As anyone got a better solution ?

Tags (3)
Highlighted

Re: Finding events that have never happened before

Communicator

It's not really "never happened before" but have you considered alert throttling? I'm not sure how long time period you can define (and will it kill the performance) but in theory something like this* could do almost what you want.

0 Karma
Highlighted

Re: Finding events that have never happened before

Communicator

Hi there, that's not really what i'm trying to do here. I need splunk to "remember" events that might have happened à couple of years ago. So i can't really afford to use alert throttling 😕

0 Karma
Highlighted

Re: Finding events that have never happened before

Legend

I'm pretty sure that there's no very fast AND easy way of doing this. There are a number of search commands that you could use for coming closer to a solution that fits you, these include anomalies, anomalousvalue and rare. The issue you will likely run into with these commands (and any other solution I can think of) is that you would have to search over all time in order to have Splunk "remember" any event that has happened before. If that's OK with you, I definitely recommend looking up all these commands and see if anyone fits your needs.

On a sidenote, I know of a pretty cool solution that does exactly this (finds "unexpected / never happened before" events in a more or less infinite time window), but it's yet to be released.

Highlighted

Re: Finding events that have never happened before

Communicator

Will try anomalies and anomalousvalue when i have a few minutes. Thanks for the tip !

What about this cool solution you are talking about ? I'm currently testing splunk 5 beta. Haven't seen such a thing though ... Can you tell me more about it ?

0 Karma
Highlighted

Re: Finding events that have never happened before

Communicator

Something to do with bloom filters maybe?

0 Karma
Highlighted

Re: Finding events that have never happened before

Legend

Not to my knowledge, no. I'm not involved myself so I can't really say anything more right now. Sorry!

0 Karma
Highlighted

Re: Finding events that have never happened before

Communicator

Please let me know when you're free to talk. I'm very interested and could not confirm the info with my contacts at Splunk.
Thanks a lot in advance !

0 Karma
Highlighted

Re: Finding events that have never happened before

Legend

Acknowledged - I'll let you know when there's anything more I can say!

0 Karma
Highlighted

Re: Finding events that have never happened before

Engager

Hi Ayn, anything new here ? (it's been a while I know)

0 Karma