topic Re: Using search count with Apache logs in Alerting

Using search count with Apache logs

tevgey23 — Tue, 10 Jul 2012 19:38:40 GMT

Im trying to use Splunk to detect a network scan using http logs. Ive been playing with the syntax
for some time but cant get it quite right.

What would the syntax be to alert on 20 server errors (i.e 503 or 404) if logged with in a 3 min window

I rather not use real time alerting as it may require more resources but im open to suggestions

Using search count with Apache logs
Thank you,

Re: Using search count with Apache logs

sideview — Tue, 10 Jul 2012 20:04:55 GMT

Actually the real-time search option usually has less practical impact on system resources than a scheduled normal search. In short the realtime search systems are dealing with events that are already moving through the system uncompressed, whereas with the normal scheduled historical search kind, the events have to be pulled off disk, gunzipped and then put together.

But I would just run the search, ie

status=50* OR status=40*

and then in the 'create alert' screens you'll be able to set a 3-minute realtime window as well as the conditions under which you'd like to get alerted.

If you do want to set it up as a historical search you still rely on the create alert workflow.
However you might want it to scan over a longer time period, and then use the search language itself to only look at 3 minute windows. One simple way is to bin the _time values to nearest minute and then just get the rolling count with streamstats.

status=50* OR status=40* | bin _time span=1min | stats count by _time | streamstats sum(count) as count window=3

Re: Using search count with Apache logs

sdaniels — Tue, 10 Jul 2012 20:05:35 GMT

It really makes sense to do this in a real time search. The example in the docs should be helpful for you to set this up.

http://docs.splunk.com/Documentation/Splunk/latest/User/Alertusecases#Alerting_when_a_set_of_IDS_solutions_report_a_network_attack_more_than_20_times_within_a_10_minute_period

Re: Using search count with Apache logs

tevgey23 — Tue, 10 Jul 2012 20:17:58 GMT

Yep I gave that a try but it seems the logs are set up a bit different

[09/Jul/2012:18:42:10 -0400] "GET/manual/ru/ja/mod/mod_status.html HTTP/1.1" 404 321 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)"

As you can see there is no "status" so 40* returns a lot of false positives.

Any other recommendations ?

Re: Using search count with Apache logs

sideview — Tue, 10 Jul 2012 20:20:20 GMT

Sorry - you will want to first extract the values as a field called 'status'. http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Addfieldsatsearchtime