Trigger on condtions met per second

[kestasm]

I am looking for possibility to be able to alert on unique source IPs within web logs, which make constant requests (GETs or POSTs) and gets server response ("403" or "407") as frequent as 20 times per 5 seconds. So for the output I want to have all the source IPs and destination matched where these conditions are met.

[martin_mueller · Accepted solution]

Are you looking for "IPs that exceeded 20 requests during some five second window" or for "IPs that average over 20 requests per five seconds (= four requests per second) over an arbitrary window"?

Assuming the latter, try this:

index=proxy http_status=403 OR http_status=407 | stats count by src_ip | addinfo | eval timerange = info_max_time - info_min_time | fields - info_* | eval rate = count / timerange | where rate > 4

The former is a bit more tricky if you want statistical accuracy. For example, it'd be easy to split each minute into twelve five-second buckets and count for each bucket. However, if you have 15 events at :04 and 15 events at :06 that'd statistically be "more than 20 in five seconds" but they'd be in different buckets so that would not trigger. If that inaccuracy is fine then use something like this:

index=proxy http_status=403 OR http_status=407 | bucket span=5s _time | stats count by _time src_ip | where count > 20

That would give you potentially splunkloads of rows per src_ip, consider adding another stats at the end if you want one row per host.

If that statistical accuracy is not fine then let me know and I'll cook up something more complicated but maybe more accurate.

[gibba]

Hi

you can use the access_combine source type like apache web log, you have automatically a field http_request for te get and post, e field source ip.
after you parsing your data you can structure your search like

"index=weblog http_request=403 http_request=407 | table source http_request"

after, click "save as" option alert

[kestasm]

Sorry I wasnt clear enough, I already have a query with conditions:
index="proxy" http_status="40*" | stats count values(host) by _time | where count > 100 | convert ctime(_time) AS time

I want to get triggered on every unique IP which exceeds (POST, GET) requests of 20 attempts per 5 seconds and only those which get "403" or "407" reply from server.