Splunk ver 7.1.1
I'm using real-time alert that trigger when there is event which has src_ip match black_list.csv like below.
index=hogehoge
| lookup black_list.csv src OUTPUT status
| where isnotnull(status)
| table _time src status
But if there is such a event, this alert is continuously triggered per 5 seconds against same event!
I think it caused by lookup
command touches every event again, when new event is arrived.
Should not I use lookup
in real-time alert?
And is it specification?
* If this is written in the document, I'm sorry ....
Please someone help me.
You probably don't need a realtime alert for this. Realtime should be reserved for ultracritical items where the SLA is seconds, rather than hours or minutes.
First, it's better to schedule a periodic scheduled search that runs every 1m, 2m, 3m or even 5m depending on what the real SLA for response is.
Second, find out what your typical delay is for indexing, and give your system at least that many minutes to process before your scan. Most installations ingest events in less than a minute, so if you're running every 3 minutes (say at 2,5,8....56,59 minutes after the hour) Then you might set the search up as
earliest=-5m@m latest=-2m@m
Third, for this particular alert, it seems to be the src
that you want to alert on, so you should put a |dedup src
or stats count by src
so that there is only one record coming out.
Fourth, you probably want to throttle the results so that, once an alert has been received for one src
, you suppress further alerts until some length of time has passed.
So, that scheduled alert would look something like this..
index=hogehoge earliest=-5m@m latest=-2m@m
| dedup src
| lookup black_list.csv src OUTPUT status
| where isnotnull(status)
| table _time src status
You will want throttle -> per result -> src
https://docs.splunk.com/Documentation/Splunk/7.1.2/Alert/ThrottleAlerts
This would trigger an alert every time something matches the lookup which is presumably 100% of the time.
what are the values of the” status” field? If they are “up/down”, then you could do this.
| where status==“down”
Instead of the where clause you have currently.