Splunk ver 7.1.1
I'm using real-time alert that trigger when there is event which has srcip match blacklist.csv like below.
index=hogehoge | lookup black_list.csv src OUTPUT status | where isnotnull(status) | table _time src status
But if there is such a event, this alert is continuously triggered per 5 seconds against same event!
I think it caused by
lookup command touches every event again, when new event is arrived.
Should not I use
lookup in real-time alert?
And is it specification?
* If this is written in the document, I'm sorry ....
Please someone help me.
This would trigger an alert every time something matches the lookup which is presumably 100% of the time.
what are the values of the” status” field? If they are “up/down”, then you could do this.
| where status==“down”
Instead of the where clause you have currently.
You probably don't need a realtime alert for this. Realtime should be reserved for ultracritical items where the SLA is seconds, rather than hours or minutes.
First, it's better to schedule a periodic scheduled search that runs every 1m, 2m, 3m or even 5m depending on what the real SLA for response is.
Second, find out what your typical delay is for indexing, and give your system at least that many minutes to process before your scan. Most installations ingest events in less than a minute, so if you're running every 3 minutes (say at 2,5,8....56,59 minutes after the hour) Then you might set the search up as
Third, for this particular alert, it seems to be the
src that you want to alert on, so you should put a
|dedup src or
stats count by src so that there is only one record coming out.
Fourth, you probably want to throttle the results so that, once an alert has been received for one
src, you suppress further alerts until some length of time has passed.
So, that scheduled alert would look something like this..
index=hogehoge earliest=-5m@m latest=-2m@m | dedup src | lookup black_list.csv src OUTPUT status | where isnotnull(status) | table _time src status
You will want throttle -> per result -> src