Splunk Search

How to perform lookup on inconsistent IPv6 format in CSV file from index?

LearningGuy
Builder

Hello,
How to perform lookup on inconsistent IPv6 format in CSV file from index?

For example:
Index has collapsed format of IPv6:  2001:db8:3333:4444:5555:6666::2101
CSV has expanded format of IPv6:    2001:db8:3333:4444:5555:6666:0:2101

The following lookup can NOT find the IPv6 that has the inconsistent pattern, it only find the exact match
| index=vulnerability_index
| lookup company.csv ip_address as ip OUTPUTNEW ip_address, company, location

In IPv6
 "::" (double colon) represents consecutive zeroes  ( :0:   or :0:0: or :0:0:0:)
 ":0:"represents 0000

I think this is what I am looking for, but I am not sure how to implement it.
https://splunkbase.splunk.com/app/4912

Thank you for your help

Labels (1)
0 Karma
1 Solution

yuanliu
SplunkTrust
SplunkTrust

I apologize for giving wrong information.  IPv6 is 128-bit, not 64 bit.  Given this lookup table and advanced option match_type CIDR(ip):

expectedip
true2001:0db8:ffff:ffff:ffff:ffff:ffff:ff00/128
test mask2001:db8:3333:4444:5555:6666::2101/128
test without mask2001:db8:3333:4444:5555:6666::2101

This search now gives the correct output

 

| makeresults
| fields - _time
| eval ip=mvappend("2001:db8:3333:4444:5555:6666:0:2101", "2001:db8:3333:4444:5555:6666::2101", "2001:0db8:ffff:ffff:ffff:ffff:ffff:ff00")
| mvexpand ip
| lookup ipv6test ip

 

expectedip
test mask2001:db8:3333:4444:5555:6666:0:2101
test mask2001:db8:3333:4444:5555:6666::2101
true2001:0db8:ffff:ffff:ffff:ffff:ffff:ff00

Hope this helps.

View solution in original post

LearningGuy
Builder

Hello,
I changed the title.  The CIDR match is used to see if a IP is within a subnet.
I was trying to match the same IPv6, but with different format from index with my CSV table.
In the example:
Index has collapsed format of IPv6:  2001:db8:3333:4444:5555:6666::2101
CSV has expanded format of IPv6:    2001:db8:3333:4444:5555:6666:0:2101
The following lookup can NOT find the IPv6 that has the inconsistent pattern, it only find the exact match
| index=vulnerability_index
| lookup company.csv ip_address as ip OUTPUTNEW ip_address, company, location

I think this is what I am looking for, I just don't know how to implement it
https://splunkbase.splunk.com/app/4912

Thank you for your help

0 Karma

yuanliu
SplunkTrust
SplunkTrust

CIDR is just a notation.  Nothing prevents you from using a 64-bit mask, i.e., host address.  For example, 2001:db8:3333:4444:5555:6666::2101/64

0 Karma

LearningGuy
Builder

Hello,
I tested by following the instruction, it only worked with the sample IP provided by Splunk, but it didn't work when I tried  compare IPv6 with IPv6.  (See below)
It looks like CIDR match only work only if a IP is part of subnet.
In my environment, I tried to compare IPv6 (compressed) with IPv6 (expanded)
Thanks for your help

IP from Splunk, expected = TRUE

| makeresults
| eval ip="2001:0db8:ffff:ffff:ffff:ffff:ffff:ff99"
| lookup ipv6test ip OUTPUT expected

_timeexpectedip
2023-09-07 09:08:54TRUE2001:0db8:ffff:ffff:ffff:ffff:ffff:ff99

 

IP from our test, expected = empty
| makeresults
| eval ip="2001:db8:3333:4444:5555:6666::2101"
| lookup ipv6test ip OUTPUT expected

_timeexpectedip
2023-09-07 09:10:54 2001:db8:3333:4444:5555:6666::2101

 

CSV Table

ipexpected
2001:0db8:ffff:ffff:ffff:ffff:ffff:ff00/120true
2001:db8:3333:4444:5555:6666::2101/64test mask
2001:db8:3333:4444:5555:6666::2101test with mask
0 Karma

yuanliu
SplunkTrust
SplunkTrust

I apologize for giving wrong information.  IPv6 is 128-bit, not 64 bit.  Given this lookup table and advanced option match_type CIDR(ip):

expectedip
true2001:0db8:ffff:ffff:ffff:ffff:ffff:ff00/128
test mask2001:db8:3333:4444:5555:6666::2101/128
test without mask2001:db8:3333:4444:5555:6666::2101

This search now gives the correct output

 

| makeresults
| fields - _time
| eval ip=mvappend("2001:db8:3333:4444:5555:6666:0:2101", "2001:db8:3333:4444:5555:6666::2101", "2001:0db8:ffff:ffff:ffff:ffff:ffff:ff00")
| mvexpand ip
| lookup ipv6test ip

 

expectedip
test mask2001:db8:3333:4444:5555:6666:0:2101
test mask2001:db8:3333:4444:5555:6666::2101
true2001:0db8:ffff:ffff:ffff:ffff:ffff:ff00

Hope this helps.

LearningGuy
Builder

Hello,
I tested your suggestion and it worked, but my CSV file does not have /128.
Can you answer the following questions?   I appreciate your help

1. a) Should I add /128 on all IPv6 on my CSV file to get this to work?
    b) If yes, does it mean I need to put extra layer to check which one is IPv6 or IPv4 and then append /128?

2. Will OUPUTNEW work just fine as regular lookup?
3. a) If I update CSV file (with new fields), will the definition lookup still work?
    b) Is there a way to automate update on the definition lookup? 
        I plan on creating automatic update on CSV, but it looks like the definition ties on specific field.

4.  Note that if I use /120, it could return multiple result like the following:

expectedip
test mask 2
test mask 4
test mask 6
2001:db8:3333:4444:5555:6666::2101

 

 

0 Karma

yuanliu
SplunkTrust
SplunkTrust

1. a) Should I add /128 on all IPv6 on my CSV file to get this to work?
    b) If yes, does it mean I need to put extra layer to check which one is IPv6 or IPv4 and then append /128?

IPv4 is 32-bit, IPv6 is 128-bit.  This means that if your CVS only contains host addresses, you need to use /128 with all IPv6 entries and /32 with all IPv4 entries.


2. Will OUPUTNEW work just fine as regular lookup?
3. a) If I update CSV file (with new fields), will the definition lookup still work?

CIDR(ip) does not change any other aspect of lookup.

3. 
    b) Is there a way to automate update on the definition lookup? 
        I plan on creating automatic update on CSV, but it looks like the definition ties on specific field.

Not sure what you mean by automation.  If you mean in the background with some external utilities, certainly.  Once lookup is defined, all you need to do is to update the file. (In distributed deployment, however, you do need to take care to update every search head.)

In Splunk, you can take a look at outputlookup.  You can use a Splunk search to update an existing lookup (even create a new one).


4.  Note that if I use /120, it could return multiple result like the following:
expectedip
test mask 2
test mask 4
test mask 6
2001:db8:3333:4444:5555:6666::2101

 


That is precisely what netmask does. (Using CIDR for host address is just a special, and less common use case.)  You can read about IP address spaces, subnet, and CIDR in a variety of online resources.

LearningGuy
Builder

Hello,

The CSV file is derived from dbxQuery, so I need to figure out how to append/128 for ipv6 and /32 for ipv6.

Does Splunk have a function to check if an IP is IPv4 or IPv6?
| dbxquery query="select IP from tableCompany"
|   eval IP = if ( isIPv4(IP),  IP=IP . "/32",  IP=IP . "/128")

Thank you so much

0 Karma

yuanliu
SplunkTrust
SplunkTrust

Splunk does not have a IP version check per se.  But you can hack ipmask to your advantage.  ipmask only works with IPv4.  So, if you are confident that your query returns legitimate IP addresses, you can tell IPv4 from IPv6.

| dbxquery query="select IP from tableCompany"
| eval IP = if(isnull(ipmask("255.255.255.255", IP)), IP . "/128", IP . "/32")

Here is a snippet to help you observe how ipmask works in this context:

| makeresults
| eval ip = mvappend("10.11.12.13", "::")
| mvexpand ip
| eval hostmask4 = ipmask("255.255.255.255", ip)

Netmask 255.255.255.255 also serves as an IPv4 validator.  IPv6 can be validated using regex, but if your database is trustworthy, you can save this trouble.

Tags (1)

yuanliu
SplunkTrust
SplunkTrust

Forget DBXquery.  Splunk's lookup can work with IPv6 CIDR.  You just need to build your lookup with CIDR.  See IPv6 CIDR match in Splunk Web (also Define a CSV lookup in Splunk Web).

Get Updates on the Splunk Community!

Index This | What are the 12 Days of Splunk-mas?

December 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Get Inspired! We’ve Got Validation that Your Hard Work is Paying Off

We love our Splunk Community and want you to feel inspired by all your hard work! Eric Fusilero, our VP of ...

What's New in Splunk Enterprise 9.4: Features to Power Your Digital Resilience

Hey Splunky People! We are excited to share the latest updates in Splunk Enterprise 9.4. In this release we ...