Why does this work: lookup, outputnew

[yoshileigh66]

Apparently my Google-Fu isn't the best and I can't find an explanation. Can someone please enlighten me? 

I have a lookup table that looks like this: 

CIDR, ip_address

24, 1.2.3.4/24
23, 5.6.7.8/23

I wanted events with source ips that match the ip addresses in the lookup table with destination ips that do not match the ip addresses in the lookup table. 
I ran the following query, and this appears to work (unless its actually not??) 

index="index1" | lookup lookup1 ip_address as src_ip OUTPUTNEW ip_address as address | where dest_ip!=address

My confusion stems from the fact that ip_address is in CIDR notation. The way my mind is processing this query is that a new field called address is created, and the value of dest_ip is compared against the value of address. However, the value of address is in CIDR notation, and dest_ip is not. 

Is address treated as a list and the value of dest_ip is checked against each item in the list? 

[yuanliu]

---

I ran the following query, and this appears to work (unless its actually not??) 

---

Your hunch is correct: That search actually does not do what you wanted because dest_ip!=address will always be true.

First things first, I assume that lookup1 has MATCH_TYPE=CIDR(ip_address) given that your search has apparent success.  As a side: as @ITWhisperer points out, the ip_address column is already in CIDR notation.  The other column (CIDR) conveys no information for this purpose.  I can't think of any use case where such a column will be useful.

---

CIDR, ip_address

24, 1.2.3.4/24
23, 5.6.7.8/23

I wanted events with source ips that match the ip addresses in the lookup table with destination ips that do not match the ip addresses in the lookup table. 

---

The second half of the statement is imprecise.  Do you mean you want dest_ip that does not match the very IP range that includes src_ip in the exact same event, or do you mean you want dest_ip that does not match any IP range in the lookup table?  If former, use a modified search from @bowesmana's answer:

index="index1" 
| lookup lookup1 ip_address as src_ip OUTPUT ip_address as address
| where isnotnull(address) AND cidrmatch(address, dest_ip)

If the latter, you can use

index="index1" 
| lookup lookup1 ip_address as src_ip OUTPUT ip_address as src_match
| where isnotnull(src_match)
| lookup lookup1 ip_address as dest_ip OUTPUT ip_address as dest_match
| where isnull(dest_match)

But your statement can have a million other interpretations.  Unless you can be specific, none of these will work for you.

[bowesmana]

You need to change your search a bit

index="index1" 
| lookup lookup1 ip_address as src_ip OUTPUT ip_address as address
| where cidrmatch(address, dest_ip)

i.e. you don't really need OUTPUTNEW unless you want to prevent an existing address field from being overwritten, but then the where clause uses the cidrmatch command where you give the CIDR range as the first parameter.

This assumes that your lookup1 is a lookup definition and you have defined the match type as CIDR(ip_address)

[ITWhisperer]

No, the number after the slash / denotes how many significant bit are to be compared when the addresses are converted to binary. There are 32 bits in the binary representation of an IPv4 address, so, for example, /24 means just compare the first 24 bits of the addresses. This is equivalent to ignoring the last byte or masking with 0xFFFFFF00, or shifting right by 8 bits (32 - 24)