Search efficiency question

[responsys_cm]

We've got a search that looks for suspicious data from a large number of netblocks. That search looks like: index=product AND src_ip="1.1.1.0/24" OR src_ip="2.2.2.0/24"...

There are dozens of them. That search is pretty fast considering the number of events we have. I would like to manage that search with a lookup table since I've created forms that make it easy to add or remove address ranges from the lookup table.

But if I search on all events from index=product and then do a lookup on that table and filter by events that match the table, the search is incredibly slow.

Is it possible to populate the src_ip=x OR src_ip=y OR src_ip=z for each entry in the lookup table? Or is there another way of implementing this that will be faster?

Thx.

Craig

[dwaddle · Accepted solution]

Well, if you have the set of src_ip values in a lookup table already, then (up to a limit) this should be doable with a subsearch and inputlookup. Assuming your lookup is formatted like this:

src_ip
1.1.1.0/24
2.2.2.0/24

then a search like this ought to work:

index=product [ | inputlookup netblocks.csv | fields src_ip ]

The subsearch will populate the outer search with a sequence of AND'ed and OR'ed values coming from the results of the subsearch. There is a limit to the subsearch, however, to around 10,000 values. This is probably a limit than can be increased some, but it may not be possible to make it arbitrarily large.

[dwaddle]

It is a function of the lookup you provide and the use of the "format" search command. Basically, each "row" of the lookup is surrounded by an OR, and each "column" within the lookup is surrounded by an AND. So, a lookup with the tuples (foo,green,4),(bar,red,7),(baz,yellow,5) will produce ( ( foo AND green AND 4 ) OR ( bar AND red AND 7 ) OR ( baz AND yellow AND 5 ) )

[responsys_cm]

How can it populate the outer search with both AND'ed and OR'ed values? Does it search for both? How can I force it to use one or the other?