IPv6 summarization - config to perform at search time

[ahartge]

I receive logs from a device with the full form IPv6 address, as well as using capital letters.

Example: 2001:0DB8:85A3:0000:0000:8A2E:0370:7334

I have a lookup table with summarised IPv6 addresses that I want to match against.

src_ip,bunit
2001:db8:85a3::8a2e:370:7334 ,bobs_burgers

Using SPL-fu I can do massage the addresses so the lookup match works

Run Anywhere Example:

| makeresults | eval src_ip="2001:0DB8:85A3:0000:0000:8A2E:0370:7334"
| eval src_ip=lower(src_ip)
| rex mode=sed field=src_ip "s/(:0{4})+/:/g"
| rex mode=sed field=src_ip "s/:(0){1,3}/:/g"

My question is how should these be applied without users having to manually add the SPL. I cannot seem to find how to add SED or lowercase transforms at search time. There is many references of SEDCMD at index time to mask / replace etc, but I want to store the original data and do this at search time.

Is there a solution to create a transform that can perform this at search time, or am I resigned to adding the 3 eval/rex lines to all searches / dashboards that use this data.

Thanks,
Ash

[woodcock]

All that you need to do is create a Lookup definition that references your existing Lookup file and inside of that definition change the Advanced options setting for Case sensitive match. Then you need not concern yourself with case.

[FrankVl]

Wait, you set the matching to case sensitive to "not concern with case"? And that doesn't solve his need to strip leading zero's and collapse full-zero blocks.

[woodcock]

Missed that part.

[somesoni2]

You can merge your lower and sed commands into single eval expression like this (src_ip field is modified your logic, src_ip1 is using new consolidated logic. Use expression from line2)

| gentimes start=-1  | eval src_ip="2001:0DB8:85A3:0000:0000:8A2E:0370:7334" | table src_ip
| eval src_ip1=lower(replace(replace(src_ip,"\:0{1,4}",":"),":::","::")) 
| eval src_ip=lower(src_ip) 
 | rex mode=sed field=src_ip "s/(:0{4})+/:/g"
 | rex mode=sed field=src_ip "s/:(0){1,3}/:/g" | eval result=if(src_ip=src_ip1,"Yes","No")

[FrankVl]

@somesoni2: that eval on line 2 would then easily translate into a props.conf EVAL, but any ideas how to prevent such config from collapsing multiple non-consecutive blocks of zeros? Also your code doesn't work well when there are more than 2 consecutive zero-blocks (the outer replace should match "::+" I guess.

Edit: had a small brainwave on the possible regex, moved my suggestion to a comment below my own answer.

[FrankVl]

First, a comments on your current SED commands: You're only allowed to collapse 1 sequence of all zero's (can span multiple blocks). E.g. you cannot collapse 2001:0DB8:85A3:0000:0000:8A2E:0000:7334 to 2001:0DB8:85A3::8A2E::7334 as that results in an ambiguous address. Your current first SED command would do that however.

To put this into props.conf, you could use an EVAL-src-ip=lower(replace(X,Y,Z)) command. See: http://docs.splunk.com/Documentation/Splunk/7.1.0/SearchReference/TextFunctions#replace.28X.2CY.2CZ....

But you would need to come up with something smart to prevent the issue of replacing multiple non-adjacent blocks of zeros.

[ahartge]

HI Frank,

Thanks so much for your continued help with this. One way to lessen the chances might be to modify the regex to match only two or more instances together, leaving single instances of 0000 alone.

| rex mode=sed field=src_ip "s/(:0{4}){2,}/:/g"

not 100% but closer.

[ahartge]

example of this is 2001:0000:85A3:0000:0000:7334:0000:0001

| makeresults | eval src_ip="2001:0000:85A3:0000:0000:7334:0000:0001"
 | eval src_ip=lower(src_ip)
 | rex mode=sed field=src_ip "s/(:0{4}){2,}/:/g"
 | rex mode=sed field=src_ip "s/:(0){1,3}/:/g"

2001:0:85a3::7334:0:1

[FrankVl]

The code I gave in my comment to my own answer should do the trick I think.

[FrankVl]

Perhaps using a transforms.conf stanza with a regex that captures an arbitrary set of random blocks in $1, then ignores a potential set of zero-blocks and then captures any remaining random blocks in $2.
https://regex101.com/r/BEifdb/2

[src-ip-collapse-ipv6]
SOURCE_KEY = src-ip
REGEX = ((?:(?:^|:)\w{0,3}[1-9A-Z]\w{0,3})*)(?:(?:(?:^|:)0000)*(?::|$))((?:$|\w{0,3}[1-9A-Z]\w{0,3})(?:$|:\w{4})*)
DEST_KEY = src-ip
FORMAT = $1::$2

And then in props.conf a REPORT to refer to transforms.conf and an eval to make it lowercase, strip 1-3 leading zeros from blocks and remove :: from an address that wasn't collapsed:

REPORT-collapsed-src-ip-from-src-ip = src-ip-collapse-ipv6
EVAL-src-ip = lower(replace(replace(src-ip,":0{1,3}",":"),"^((?:\w{4}:){7}\w{4})::$","$1")

[ssadanala1]

settings --> Caluclated fields is the solution for this .

The documentation is pretty straight forward

https://docs.splunk.com/Documentation/Splunk/latest/Knowledge/definecalcfields

Happy Splunking !!

[ahartge]

this is one reason why I love splunk answers - feeling a little silly now lol - thanks for the link.