Splunk Search

Regex for extracting ip port and interface

jalfrey
Communicator

In my log data I get lines that look like this:
dst=10.0.59.59:80:X1
dst=255.255.255.255:67:X0
dst=10.0.59.59:9060:X1
dst=0.0.0.0:0:X0
dst=224.0.0.5:1

The first value is an IP address. The next two values should be port number and interface. I did some field extractions and I can get it to extract all three fields if they are present but when it has only 2 fields it throws away the data. Could you help me write a field extraction regex to get the 2 field and 3 field variants.

the field names should be dst_ip, dst_port, dst_interface

0 Karma
1 Solution

kristian_kolb
Ultra Champion

Well, since there might not always be three fields to extract from the same piece of data, I'd do it in two EXTRACTs in props.conf

[your sourcetype]
EXTRACT-dst_ip_port = \s+dst=(?<dst_ip>[^:]+):(?<dst_port>\d+)
EXTRACT-dst_if = \s+dst=[^:]+:\d+:(?<dst_interface>\S+)

/K

View solution in original post

kristian_kolb
Ultra Champion

that's why I put in \S+, i.e. 'one or more non-whitespace characters' for the interface extraction. If this indeed captures more than intended you should post a couple of full events, or explain more clearly how the fields are delimited in your events.

0 Karma

bmacias84
Champion

I would use a transform for this one. I not sure how you event is seperated, this is for space delim event. Using Transforms you can create additional extraction from an already extracted value.

example Event : 2012-04-23 13:24:25 SUCCESS 10.0.59.59:9060:2561X0 10.214.1.79:9060:X1

First trasnfrom does the following key value pairs from _raw:

  • date=2012-04-23
  • time=13:24:25
  • status=SUCCESS
  • scr=10.0.59.59:9060:2561X0
  • dst=10.214.1.79:9060:X1

The second transform does the following by using dst as the source_key to prefrom extraction.

  • ip=10.214.1.79
  • port=9060
  • interface=X1
    
    #transforms.conf
    [some_event]
    DELIMS = " "
    FIELDS = date,time,status,scr,dst
    [dstextract]
    SOURCE_KEY = dst
    DELIM = ":"
    FIELDS = ip,port,interface
    
    
    #props.conf
    [Mysource]
    MAX_TIMESTAMP_LOOKAHEAD=40
    NO_BINARY_CHECK=1
    SHOULD_LINEMERGE=false
    TZ=US/Pacific
    REPORT-Mysource=some_event,dstextract
    

Hope this help or gives you some ideas. Dont forget to accept and vote up answers that up.

Cheers,

0 Karma

bmacias84
Champion

I preform all configurations through direct edit of the conf files. The GUI only provides limited functionality for advanced configurations direct editing is required.

0 Karma

jalfrey
Communicator

this looks good. I see that you posted to do it in flat text. Is there any way to do this in the GUI so I can test the functionality?

0 Karma

jalfrey
Communicator

for searching for dst_ip and dst_port this seems to work
(?i) dst=(?P\d+.\d+.\d+.\d+):(?P[0-9]+)
No idea if that's efficient

0 Karma

jalfrey
Communicator

this one is a little better
(?i) dst=[^:]+:\d+:(?P\w[0-9]+)
I found out we have other things then just X and possibly 2 digits

0 Karma

jalfrey
Communicator

oh this seems to work
(?i) dst=[^:]+:\d+:(?PX[0-9])

0 Karma

kristian_kolb
Ultra Champion

Well, since there might not always be three fields to extract from the same piece of data, I'd do it in two EXTRACTs in props.conf

[your sourcetype]
EXTRACT-dst_ip_port = \s+dst=(?<dst_ip>[^:]+):(?<dst_port>\d+)
EXTRACT-dst_if = \s+dst=[^:]+:\d+:(?<dst_interface>\S+)

/K

jalfrey
Communicator

yes you must do a splunk restart

0 Karma

jalfrey
Communicator

when I change the props.conf do I need to restart splunk?

0 Karma

kristian_kolb
Ultra Champion

The regexes I listed above are meant to go into props.conf.

Also, when posting, use the backtick (`) around code examples. otherwise a lot of stuff will be filtered out.

0 Karma

jalfrey
Communicator

when I remove the "dst=" and add colons to be beginning and end so it looks like this:
:10.0.59.59:80:X1:

The regex generated looks like:
(?:[^ \n]* ){3}(?P[^ ]+)(?:[^.\n].){5}\d+(?P\d+)(?:[^:\n]:){2}(?P[^ ]+)

When I put your regex into the Interactive field extractor I get nothing. Does the \s+dst need a different beginning?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...