Splunk Search

Finding events that have never happened before

Mahieu
Communicator

Hello there,

I'm pretty someone has asked the question before but couldn't find the post.
I'm trying to find a good way to search for events that have never happened before (for alerting purposes).

I've got two options.
Option 1 is based on using the punct field. I'll do something like rare punct | where punct <x and i'll be able to see the logs that have "weird" patterns. Problem is, if a log has the same format than another one, i won't be able to detect it as "never happened before".
Option 2 is based on a succession of "NOT" that would exclude everything i know from the results. This search takes a while to settle but that's fine, i've got time 😉

In both cases, when my search will generate a result i'll either :
- update the search so that the event does not generate another alert in the future (if it's considered "safe")
- leave the search this way as i'm interested in detected this kind of events

As anyone got a better solution ?

Tags (3)

Runals
Motivator

Little old but ran across this while searching for something else. Here is how I would attempt to tackle this.

Obviously a number of complexities based on your environment but I would look at a two stage approach. The first is create a scheduled search that outputs a csv containing known events. The second stage is to create your alert (again scheduled) with a subsearch that builds your NOT statement. Take Windows events for example. In my first scheduled search I might build my csv that contains EventCode, host, date first seen, sum of events since first seen, and most recent time of the EventCode & host pair. Schedule that to run once or twice a day (or whatever). Let that bake for a scheduled run or two.

The second stage is to create your alert. To extend the Windows example you might create your query like

index=whatever sourcetype=wineventlog* NOT [ | inputlookup known_events.csv | fields host EventCode] | ...

The 'trick' is to figure out what sort of thresholds your are comfortable with. The primary driver to me is how often you want the second search to run. In other words if only want that to run once a day then you probably only need the first search to run once per day as well (scheduled to finish before the second starts). If you wanted the second search to run once every hour you could still have the first search run once a day but there is a potential of getting 23 alerts (ie., a new system starts sending data).

0 Karma

mcm10285
Communicator

If you have not found a solution yet, there's an app called prelert in splunkbase.

0 Karma

Ayn
Legend

I'm pretty sure that there's no very fast AND easy way of doing this. There are a number of search commands that you could use for coming closer to a solution that fits you, these include anomalies, anomalousvalue and rare. The issue you will likely run into with these commands (and any other solution I can think of) is that you would have to search over all time in order to have Splunk "remember" any event that has happened before. If that's OK with you, I definitely recommend looking up all these commands and see if anyone fits your needs.

On a sidenote, I know of a pretty cool solution that does exactly this (finds "unexpected / never happened before" events in a more or less infinite time window), but it's yet to be released.

Nomios
Engager

Hi Ayn, anything new here ? (it's been a while I know)

0 Karma

Ayn
Legend

Acknowledged - I'll let you know when there's anything more I can say!

0 Karma

Mahieu
Communicator

Please let me know when you're free to talk. I'm very interested and could not confirm the info with my contacts at Splunk.
Thanks a lot in advance !

0 Karma

Ayn
Legend

Not to my knowledge, no. I'm not involved myself so I can't really say anything more right now. Sorry!

0 Karma

kallu
Communicator

Something to do with bloom filters maybe?

0 Karma

Mahieu
Communicator

Will try anomalies and anomalousvalue when i have a few minutes. Thanks for the tip !

What about this cool solution you are talking about ? I'm currently testing splunk 5 beta. Haven't seen such a thing though ... Can you tell me more about it ?

0 Karma

Mahieu
Communicator

Hi there, that's not really what i'm trying to do here. I need splunk to "remember" events that might have happened à couple of years ago. So i can't really afford to use alert throttling 😕

0 Karma

kallu
Communicator

It's not really "never happened before" but have you considered alert throttling? I'm not sure how long time period you can define (and will it kill the performance) but in theory something like this* could do almost what you want.

0 Karma
Get Updates on the Splunk Community!

This Week's Community Digest - Splunk Community Happenings [9.26.22]

Get the latest news and updates from the Splunk Community here! Upcoming User Group Events! &#x1f44f; Check ...

BSides Splunk 2022 - The Call for Papers is now Open!

TLDR; Main Site: https://bsidessplunk.com CFP Site: https://bsidessplunk.com/cfp CFP Opens: December 15th, ...

Sending Metrics to Splunk Enterprise With the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...