topic Re: A more efficient query ? in Splunk Search

A more efficient query ?

ddebevec — Mon, 03 Dec 2012 21:05:28 GMT

One of our users has beought forth the following question:

I would like to be able to determine if IP Addresses from China are attempting to hit one or more of our servers. I have a list IP ranges for China in CIDR Notation, to the tune of 3400+, ranges. I have figured out how to make a query using the cidrmatch function, but I am afraid that a query of this nature may cause a severe negative impact on the performance of our Splunk environment.

The basics of the query that I have put together are:

host=myhost AND error_code-12345 | where (cidrmatch("cidrblk1/24", src) OR cidrmatch("cidrblk2/17", src) OR cidrmatch("cidrblk3/19", src) OR...)

Is there a better, more efficient way, "SAFER" way to handle this?

Is there a better way to handle this?

Re: A more efficient query ?

Ayn — Mon, 03 Dec 2012 21:24:07 GMT

The search command can handle CIDR blocks directly, so there's no need to use a separate where clause there. That will also speed things up.

host=myhost AND error_code=12345 AND (src=cidrblk1/24 OR src=cidrblk2/17 OR ...)

Note also that the MAXMIND app is pretty good for when you want to perform geolocation searches like this. You'll find it here: http://splunk-base.splunk.com/apps/22282/geo-location-lookup-script-powered-by-maxmind

Re: A more efficient query ?

cmbarber — Mon, 03 Dec 2012 22:42:00 GMT

Ayn, Thanks for the reply. I am actually the per who asked this question of ddebevec. That will shortent eh actual text of the query. I am looking to see if this is the mst efficient way to do this, or is there another means? The query would still have 3400+ items. Would something like the following severly impact the performance of the server?

host=myhost AND error_code=12345 AND (src=cidrblk1/24 OR src=cidrblk2/17 OR ... src=cidrblk1142/24 OP ... OR src=cidrblk2237/15 OR ... OR src=cidrblk3447/23)

Is ther ea way to use a table or some other construct?

Re: A more efficient query ?

Ayn — Mon, 03 Dec 2012 22:51:07 GMT

It will not just shorten it, most of all it will make it faster, because the search command will not have to retrieve as many items from disk just to send them off to where which will just discard lots of them.

If you have that many CIDR blocks, I'd advise you to use a lookup table (they can do CIDR matches as well) - not that I think it'll make any difference performance-wise, but the query will obviously not become two miles long 🙂