topic Re: Regex for extracting ip port and interface in Splunk Search

Regex for extracting ip port and interface

jalfrey — Mon, 28 Sep 2020 13:55:33 GMT

In my log data I get lines that look like this:
dst=10.0.59.59:80:X1
dst=255.255.255.255:67:X0
dst=10.0.59.59:9060:X1
dst=0.0.0.0:0:X0
dst=224.0.0.5:1

The first value is an IP address. The next two values should be port number and interface. I did some field extractions and I can get it to extract all three fields if they are present but when it has only 2 fields it throws away the data. Could you help me write a field extraction regex to get the 2 field and 3 field variants.

the field names should be dst_ip, dst_port, dst_interface

Re: Regex for extracting ip port and interface

kristian_kolb — Thu, 16 May 2013 21:03:04 GMT

Well, since there might not always be three fields to extract from the same piece of data, I'd do it in two EXTRACTs in props.conf

[your sourcetype]
EXTRACT-dst_ip_port = \s+dst=(?<dst_ip>[^:]+):(?<dst_port>\d+)
EXTRACT-dst_if = \s+dst=[^:]+:\d+:(?<dst_interface>\S+)

Re: Regex for extracting ip port and interface

jalfrey — Thu, 16 May 2013 21:15:16 GMT

when I remove the "dst=" and add colons to be beginning and end so it looks like this:
:10.0.59.59:80:X1:

The regex generated looks like:
(?:[^ \n]* ){3}(?P[^ ]+)(?:[^.\n].){5}\d+(?P\d+)(?:[^:\n]:){2}(?P[^ ]+)

When I put your regex into the Interactive field extractor I get nothing. Does the \s+dst need a different beginning?

Re: Regex for extracting ip port and interface

jalfrey — Thu, 16 May 2013 21:24:59 GMT

oh this seems to work
(?i) dst=[^:]+:\d+:(?PX[0-9])

Re: Regex for extracting ip port and interface

jalfrey — Thu, 16 May 2013 21:42:22 GMT

this one is a little better
(?i) dst=[^:]+:\d+:(?P\w[0-9]+)
I found out we have other things then just X and possibly 2 digits

Re: Regex for extracting ip port and interface

jalfrey — Mon, 28 Sep 2020 13:55:36 GMT

for searching for dst_ip and dst_port this seems to work
(?i) dst=(?P\d+.\d+.\d+.\d+):(?P[0-9]+)
No idea if that's efficient

Re: Regex for extracting ip port and interface

bmacias84 — Thu, 16 May 2013 21:49:54 GMT

I would use a transform for this one. I not sure how you event is seperated, this is for space delim event. Using Transforms you can create additional extraction from an already extracted value.

example Event : 2012-04-23 13:24:25 SUCCESS 10.0.59.59:9060:2561X0 10.214.1.79:9060:X1

First trasnfrom does the following key value pairs from _raw:

date=2012-04-23
time=13:24:25
status=SUCCESS
scr=10.0.59.59:9060:2561X0
dst=10.214.1.79:9060:X1

The second transform does the following by using dst as the source_key to prefrom extraction.

ip=10.214.1.79
port=9060

interface=X1


#transforms.conf
[some_event]
DELIMS = " "
FIELDS = date,time,status,scr,dst
[dstextract]
SOURCE_KEY = dst
DELIM = ":"
FIELDS = ip,port,interface


#props.conf
[Mysource]
MAX_TIMESTAMP_LOOKAHEAD=40
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=false
TZ=US/Pacific
REPORT-Mysource=some_event,dstextract

Hope this help or gives you some ideas. Dont forget to accept and vote up answers that up.

Cheers,

Re: Regex for extracting ip port and interface

kristian_kolb — Thu, 16 May 2013 21:52:08 GMT

that's why I put in \S+, i.e. 'one or more non-whitespace characters' for the interface extraction. If this indeed captures more than intended you should post a couple of full events, or explain more clearly how the fields are delimited in your events.

Re: Regex for extracting ip port and interface

kristian_kolb — Thu, 16 May 2013 21:55:37 GMT

The regexes I listed above are meant to go into props.conf.

Also, when posting, use the backtick (`) around code examples. otherwise a lot of stuff will be filtered out.

Re: Regex for extracting ip port and interface

jalfrey — Thu, 16 May 2013 22:48:06 GMT

this looks good. I see that you posted to do it in flat text. Is there any way to do this in the GUI so I can test the functionality?

Re: Regex for extracting ip port and interface

bmacias84 — Thu, 16 May 2013 23:00:25 GMT

I preform all configurations through direct edit of the conf files. The GUI only provides limited functionality for advanced configurations direct editing is required.

Re: Regex for extracting ip port and interface

jalfrey — Mon, 17 Jun 2013 20:15:15 GMT

when I change the props.conf do I need to restart splunk?

Re: Regex for extracting ip port and interface

jalfrey — Mon, 17 Jun 2013 20:19:01 GMT

yes you must do a splunk restart