I have a csv with ip addresses. I would like to conduct a search for addresses that are NOT listed in that csv.
I was attempting the following but it does not render the results I was expecting. I want to search for ip addresses that are not in that list. IE: unknown address...
index=myindex
| rex "(?<ip>\b(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b)" | sort ip | table ip NOT [inputlookup known_addresses.csv]
inputlookup is the wrong approach to finding events that do not have a match. (Unless you don't have a choice, or if it is applicable in index search, inputlookup is usually wrong.) lookup command is way more efficient.
index=myindex
| rex "(?<ip>\b(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b)"
| lookup known_addresses.csv ip output ip as match
| where isnull(match)
| sort ip
| table ip
I recommend first running a search using only inputlookup to ensure that your IP addresses are returning properly:
| inputlookup known_addresses.csv
You should get a single column of addresses with the "ip" field name.
ip |
192.168.1.1 |
123.123.123.123 |
222.111.133.111 |
Then you can put it into a negated search filter in your main search: (I haven't checked your regex, so assume it works to create a field of "ip" with an ip address value.)
index=myindex
| rex field=_raw "(?<ip>\b(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b)"
| search NOT [| inputlookup known_addresses.csv]
| sort ip
| table ip
If the regex does not work, you can try this one:
index=myindex
| rex field=_raw "(?<ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})"
| search NOT [| inputlookup known_addresses.csv]
| sort ip
| table ip
You may also want to put dedup at the end, to remove duplicate ip addresses:
...
| dedup ip