topic Dedup vs. Lookup performance in Splunk Search

Dedup vs. Lookup performance

responsys_cm — Fri, 27 Jul 2012 18:21:14 GMT

I have some very high volume firewall records. I want to check the destination IP address against a lookup table that contains known malware C&C IPs.

Is it more efficient to dedup the records and then do the lookup or is it faster to do a lookup on each one?

Thx.

Craig

Re: Dedup vs. Lookup performance

dwaddle — Fri, 27 Jul 2012 19:47:28 GMT

Part of it will be depend on the size of the lookup table. Neither choice has a nonzero cost. Dedup'ing can be expensive for a large number of events, but comparing against a large lookup table might also be expensive.

I would recommend that you model out both scenarios and use the search job inspector to compare and contrast the amount of time spent in each.

Also, you may want to consider summary indexing as a 3rd alternative. A summary that gets updated every few minutes on something as simple as | sistats count by destip could give you a workably fast solution.

Re: Dedup vs. Lookup performance

ziegfried — Sat, 28 Jul 2012 01:10:11 GMT

Additionally, if the lookup table is small enough, you could use inputlookup in a subsearch to query only addresses from the lookup list:

sourcetype=myfirewall [ | inputlookup cc_ipslist | return 10000 ip ]

will expand to smth like

sourcetype=myfirewall ( ip=1.1.1.1 OR ip=2.2.2.2 OR ip=3.3.3.3 ....)