Archive

Ideas on how to handle event flood/storm

ananth_nag_kavu
Explorer

We occasionally receive hundreds of thousands of events (sometimes millions) from one or two hosts and if not acted quickly, can potentially trip the indexing limit. Is there a way, built-in/otherwise, for Splunk to recognize the storm, auto-suppress these events and alert saying that "received more than K events in the last mins -- suppressing further similar events from the host for the next mins"?

Thanks!

Tags (1)
0 Karma

jkat54
SplunkTrust
SplunkTrust

Can you do something in props or transforms to discard the events such as whitelisting or blacklisting? I'm assuming the events are the same in some manor. ie. EventCode=5000

0 Karma

ananth_nag_kavu
Explorer

Yes and I am currently suppressing these flood events. Unfortunately though, sometimes it can be very late by the time the flood events can be suppressed and they may already have taken up a good chunk of your daily license limit, if they haven't already tripped it. I am hoping if Splunk/some Splunk-app can auto-learn/recognize the flood storm and temporarily suppress these events and alert the admin, so they can take a look. In short, a a built-in flood protection.

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!