Splunk Search
Highlighted

In real-time alert, if I use lookup command, too many alerts triggered.

Builder

Splunk ver 7.1.1

I'm using real-time alert that trigger when there is event which has srcip match blacklist.csv like below.

index=hogehoge 
| lookup black_list.csv src OUTPUT status
| where isnotnull(status) 
| table _time src status

But if there is such a event, this alert is continuously triggered per 5 seconds against same event!
I think it caused by lookup command touches every event again, when new event is arrived.

Should not I use lookup in real-time alert?
And is it specification?
* If this is written in the document, I'm sorry ....

Please someone help me.

0 Karma
Highlighted

Re: In real-time alert, if I use lookup command, too many alerts triggered.

SplunkTrust
SplunkTrust

This would trigger an alert every time something matches the lookup which is presumably 100% of the time.

what are the values of the” status” field? If they are “up/down”, then you could do this.

| where status==“down”

Instead of the where clause you have currently.

0 Karma
Highlighted

Re: In real-time alert, if I use lookup command, too many alerts triggered.

SplunkTrust
SplunkTrust

You probably don't need a realtime alert for this. Realtime should be reserved for ultracritical items where the SLA is seconds, rather than hours or minutes.

First, it's better to schedule a periodic scheduled search that runs every 1m, 2m, 3m or even 5m depending on what the real SLA for response is.

Second, find out what your typical delay is for indexing, and give your system at least that many minutes to process before your scan. Most installations ingest events in less than a minute, so if you're running every 3 minutes (say at 2,5,8....56,59 minutes after the hour) Then you might set the search up as

earliest=-5m@m latest=-2m@m

Third, for this particular alert, it seems to be the src that you want to alert on, so you should put a |dedup src or stats count by src so that there is only one record coming out.

Fourth, you probably want to throttle the results so that, once an alert has been received for one src, you suppress further alerts until some length of time has passed.

So, that scheduled alert would look something like this..

 index=hogehoge earliest=-5m@m latest=-2m@m
 | dedup src
 | lookup black_list.csv src OUTPUT status
 | where isnotnull(status) 
 | table _time src status

You will want throttle -> per result -> src

https://docs.splunk.com/Documentation/Splunk/7.1.2/Alert/ThrottleAlerts

0 Karma