Dedup vs. Lookup performance

[responsys_cm]

I have some very high volume firewall records. I want to check the destination IP address against a lookup table that contains known malware C&C IPs.

Is it more efficient to dedup the records and then do the lookup or is it faster to do a lookup on each one?

Thx.

Craig

[ziegfried]

Additionally, if the lookup table is small enough, you could use inputlookup in a subsearch to query only addresses from the lookup list:

sourcetype=myfirewall [ | inputlookup cc_ipslist | return 10000 ip ]

will expand to smth like

sourcetype=myfirewall ( ip=1.1.1.1 OR ip=2.2.2.2 OR ip=3.3.3.3 ....)

[dwaddle · Accepted solution]

Part of it will be depend on the size of the lookup table. Neither choice has a nonzero cost. Dedup'ing can be expensive for a large number of events, but comparing against a large lookup table might also be expensive.

I would recommend that you model out both scenarios and use the search job inspector to compare and contrast the amount of time spent in each.

Also, you may want to consider summary indexing as a 3rd alternative. A summary that gets updated every few minutes on something as simple as | sistats count by destip could give you a workably fast solution.