How to perform lookup on inconsistent IPv6 format in CSV file from index?

[LearningGuy]

Hello,
How to perform lookup on inconsistent IPv6 format in CSV file from index?

For example:
Index has collapsed format of IPv6:  2001:db8:3333:4444:5555:6666::2101
CSV has expanded format of IPv6:    2001:db8:3333:4444:5555:6666:0:2101

The following lookup can NOT find the IPv6 that has the inconsistent pattern, it only find the exact match
| index=vulnerability_index
| lookup company.csv ip_address as ip OUTPUTNEW ip_address, company, location

In IPv6
 "::" (double colon) represents consecutive zeroes  ( :0:   or :0:0: or :0:0:0:)
 ":0:"represents 0000

I think this is what I am looking for, but I am not sure how to implement it.
https://splunkbase.splunk.com/app/4912

Thank you for your help

[LearningGuy]

Hello,
I changed the title.  The CIDR match is used to see if a IP is within a subnet.
I was trying to match the same IPv6, but with different format from index with my CSV table.
In the example:
Index has collapsed format of IPv6:  2001:db8:3333:4444:5555:6666::2101
CSV has expanded format of IPv6:    2001:db8:3333:4444:5555:6666:0:2101
The following lookup can NOT find the IPv6 that has the inconsistent pattern, it only find the exact match
| index=vulnerability_index
| lookup company.csv ip_address as ip OUTPUTNEW ip_address, company, location

I think this is what I am looking for, I just don't know how to implement it
https://splunkbase.splunk.com/app/4912

Thank you for your help

[yuanliu]

CIDR is just a notation.  Nothing prevents you from using a 64-bit mask, i.e., host address.  For example, 2001:db8:3333:4444:5555:6666::2101/64

[LearningGuy]

Hello,
I tested by following the instruction, it only worked with the sample IP provided by Splunk, but it didn't work when I tried  compare IPv6 with IPv6.  (See below)
It looks like CIDR match only work only if a IP is part of subnet.
In my environment, I tried to compare IPv6 (compressed) with IPv6 (expanded)
Thanks for your help

IP from Splunk, expected = TRUE

| makeresults
| eval ip="2001:0db8:ffff:ffff:ffff:ffff:ffff:ff99"
| lookup ipv6test ip OUTPUT expected

_time	expected	ip
2023-09-07 09:08:54	TRUE	2001:0db8:ffff:ffff:ffff:ffff:ffff:ff99

IP from our test, expected = empty
| makeresults
| eval ip="2001:db8:3333:4444:5555:6666::2101"
| lookup ipv6test ip OUTPUT expected

_time	expected	ip
2023-09-07 09:10:54		2001:db8:3333:4444:5555:6666::2101

CSV Table

ip	expected
2001:0db8:ffff:ffff:ffff:ffff:ffff:ff00/120	true
2001:db8:3333:4444:5555:6666::2101/64	test mask
2001:db8:3333:4444:5555:6666::2101	test with mask

[yuanliu · Accepted solution]

I apologize for giving wrong information.  IPv6 is 128-bit, not 64 bit.  Given this lookup table and advanced option match_type CIDR(ip):

expected	ip
true	2001:0db8:ffff:ffff:ffff:ffff:ffff:ff00/128
test mask	2001:db8:3333:4444:5555:6666::2101/128
test without mask	2001:db8:3333:4444:5555:6666::2101

This search now gives the correct output

| makeresults
| fields - _time
| eval ip=mvappend("2001:db8:3333:4444:5555:6666:0:2101", "2001:db8:3333:4444:5555:6666::2101", "2001:0db8:ffff:ffff:ffff:ffff:ffff:ff00")
| mvexpand ip
| lookup ipv6test ip

expected	ip
test mask	2001:db8:3333:4444:5555:6666:0:2101
test mask	2001:db8:3333:4444:5555:6666::2101
true	2001:0db8:ffff:ffff:ffff:ffff:ffff:ff00

Hope this helps.

[LearningGuy]

Hello,
I tested your suggestion and it worked, but my CSV file does not have /128.
Can you answer the following questions?   I appreciate your help

1. a) Should I add /128 on all IPv6 on my CSV file to get this to work?
    b) If yes, does it mean I need to put extra layer to check which one is IPv6 or IPv4 and then append /128?

2. Will OUPUTNEW work just fine as regular lookup?
3. a) If I update CSV file (with new fields), will the definition lookup still work?
    b) Is there a way to automate update on the definition lookup? 
        I plan on creating automatic update on CSV, but it looks like the definition ties on specific field.

4.  Note that if I use /120, it could return multiple result like the following:

expected	ip
test mask 2 test mask 4 test mask 6	2001:db8:3333:4444:5555:6666::2101

[yuanliu]

---

1. a) Should I add /128 on all IPv6 on my CSV file to get this to work?
    b) If yes, does it mean I need to put extra layer to check which one is IPv6 or IPv4 and then append /128?

---

IPv4 is 32-bit, IPv6 is 128-bit.  This means that if your CVS only contains host addresses, you need to use /128 with all IPv6 entries and /32 with all IPv4 entries.

---

2. Will OUPUTNEW work just fine as regular lookup?
3. a) If I update CSV file (with new fields), will the definition lookup still work?

---

CIDR(ip) does not change any other aspect of lookup.

3. 
    b) Is there a way to automate update on the definition lookup? 
        I plan on creating automatic update on CSV, but it looks like the definition ties on specific field.

---

Not sure what you mean by automation.  If you mean in the background with some external utilities, certainly.  Once lookup is defined, all you need to do is to update the file. (In distributed deployment, however, you do need to take care to update every search head.)

In Splunk, you can take a look at outputlookup.  You can use a Splunk search to update an existing lookup (even create a new one).

---

4.  Note that if I use /120, it could return multiple result like the following:

expected	ip
test mask 2 test mask 4 test mask 6	2001:db8:3333:4444:5555:6666::2101

---

That is precisely what netmask does. (Using CIDR for host address is just a special, and less common use case.)  You can read about IP address spaces, subnet, and CIDR in a variety of online resources.

[LearningGuy]

Hello,

The CSV file is derived from dbxQuery, so I need to figure out how to append/128 for ipv6 and /32 for ipv6.

Does Splunk have a function to check if an IP is IPv4 or IPv6?
| dbxquery query="select IP from tableCompany"
|   eval IP = if ( isIPv4(IP),  IP=IP . "/32",  IP=IP . "/128")

Thank you so much

[yuanliu]

Splunk does not have a IP version check per se.  But you can hack ipmask to your advantage.  ipmask only works with IPv4.  So, if you are confident that your query returns legitimate IP addresses, you can tell IPv4 from IPv6.

| dbxquery query="select IP from tableCompany"
| eval IP = if(isnull(ipmask("255.255.255.255", IP)), IP . "/128", IP . "/32")

Here is a snippet to help you observe how ipmask works in this context:

| makeresults
| eval ip = mvappend("10.11.12.13", "::")
| mvexpand ip
| eval hostmask4 = ipmask("255.255.255.255", ip)

Netmask 255.255.255.255 also serves as an IPv4 validator.  IPv6 can be validated using regex, but if your database is trustworthy, you can save this trouble.

[yuanliu]

Forget DBXquery.  Splunk's lookup can work with IPv6 CIDR.  You just need to build your lookup with CIDR.  See IPv6 CIDR match in Splunk Web (also Define a CSV lookup in Splunk Web).