Splunk Search

In real-time alert, if I use lookup command, too many alerts triggered.

yutaka1005
Builder

Splunk ver 7.1.1

I'm using real-time alert that trigger when there is event which has src_ip match black_list.csv like below.

index=hogehoge 
| lookup black_list.csv src OUTPUT status
| where isnotnull(status) 
| table _time src status

But if there is such a event, this alert is continuously triggered per 5 seconds against same event!
I think it caused by lookup command touches every event again, when new event is arrived.

Should not I use lookup in real-time alert?
And is it specification?
* If this is written in the document, I'm sorry ....

Please someone help me.

0 Karma

DalJeanis
Legend

You probably don't need a realtime alert for this. Realtime should be reserved for ultracritical items where the SLA is seconds, rather than hours or minutes.

First, it's better to schedule a periodic scheduled search that runs every 1m, 2m, 3m or even 5m depending on what the real SLA for response is.

Second, find out what your typical delay is for indexing, and give your system at least that many minutes to process before your scan. Most installations ingest events in less than a minute, so if you're running every 3 minutes (say at 2,5,8....56,59 minutes after the hour) Then you might set the search up as

earliest=-5m@m latest=-2m@m

Third, for this particular alert, it seems to be the src that you want to alert on, so you should put a |dedup src or stats count by src so that there is only one record coming out.

Fourth, you probably want to throttle the results so that, once an alert has been received for one src, you suppress further alerts until some length of time has passed.

So, that scheduled alert would look something like this..

 index=hogehoge earliest=-5m@m latest=-2m@m
 | dedup src
 | lookup black_list.csv src OUTPUT status
 | where isnotnull(status) 
 | table _time src status

You will want throttle -> per result -> src

https://docs.splunk.com/Documentation/Splunk/7.1.2/Alert/ThrottleAlerts

0 Karma

jkat54
SplunkTrust
SplunkTrust

This would trigger an alert every time something matches the lookup which is presumably 100% of the time.

what are the values of the” status” field? If they are “up/down”, then you could do this.

| where status==“down”

Instead of the where clause you have currently.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...