OK - so I'm quickly becoming an even bigger Splunk fan when I put some of our key web access logs there to analyze what is going on when certain events impact the web servers. I was able to easily create reports and dashboards to view what is hitting the servers, identify hotspots by client or URI across a number of very disjointed web servers. Great so far.
But, now that I have it there I should be able to also alert on certain indicators that something bad has been logged in terms of what is in the access requests? I know that this is spilling into the realm of IDS and we don't want to duplicate work. But, our IDS will not detect "bad" requests in ssl (at least not without putting the key in so it can - which our does not have). So, If some "bad" access makes it to our server without the IDS detecting it but my access log is going to Splunk, couldn't I alert on it there.
Here is where my brilliant idea falls short - I'm not sure how to go about it. What am I looking for?
Has anyone tried to do anything like this?
I'd love to here your method and resuits.
Think of it this way... Splunk can tell you when some thing "bad" has happened, because you already know what "good" looks like. So you use your knowledge of what the web servers are supposed to be doing and you use Splunk to show you what's "normal"... and then you look for what is NOT normal. As you said, very similar to IDS where you have patterns that are recognized as "okay" so as to identify things that are not.
But using Splunk you can be less specific and more creative.
Things like, excessively long URI's or URI's that aren't formatted as you expect. Or missing useragents... or useragents that are way too small.... or way too long...
You can't be too specific with things like that because we know that the whole concept of "useragent" is a wild west out there... but a one word or missing useragent is a surefire way to identify something not awesome.
Another good one is to observe the clientip and see if you've had successive hits from clientips in the same CIDR, ie someone is banging away with a legit URI, but the ip is cycling...
There are some good web-based searches on http://gosplunk.com (mostly IIS, but you can extrapolate to other flavors of web access files) that will get you started.
It would be great if folks start posting their methodologies... but I think it will help to kind of wrap your head around the idea that if you know what "normal" is... be it the expected rapidity of consecutive hits on certain pages (person vs. script/robot) or things that are just way off based on what the values should "look" like - I'm sure you'll come up with some great stuff.
A wonderful experiment would be to check out the settings of the IDS stuff and duplicate it in Splunk as a failsafe. That will give you an idea as to whether the 'rules' are too strict for their own good or not.
With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!