topic Field extraction using regex -- Do I need to use an if or eval for this? in Splunk Search

Field extraction using regex -- Do I need to use an if or eval for this?

dipasqum — Tue, 12 Dec 2017 21:10:47 GMT

I'm a newbie/rusty w/ regex and I'm sure I'm over thinking this. I have lines from a firewall log and I would like to extract a field called protocol. I want the "TCP" and "ICMP" from the following two lines where the second line has a second word so its a little more complex. Do I use an if() or an eval to do this or is it simpler than that?

Teardown TCP connection 3119818 for outside:209.98.44.193/49936 to inside:192.168.1.8/1547 duration 0:05:01 bytes 931 TCP Reset-I

Built outbound ICMP connection for faddr 192.168.151.96/512 gaddr 192.168.1.1/0 laddr 192.168.1.1/0

Re: Field extraction using regex -- Do I need to use an if or eval for this?

cpetterborg — Wed, 13 Dec 2017 02:27:41 GMT

rex is a command so no need for if or eval. Try:

rex "\s(?P<protocol>(TCP|ICMP))\s"

Re: Field extraction using regex -- Do I need to use an if or eval for this?

dipasqum — Wed, 13 Dec 2017 17:48:32 GMT

thanks cpetterborg but I don't want this hardcoded because then it would miss protocols that could be there but aren't in my sample.

^(?:[^:\n]*:){6}\s+\w+\w+\s+[a-z]+\s(?P\w+) is something I tried. What is in bold is the content thats needed to extract into the field.

Sep 7 15:47:28 10.1.1.99 Sep 07 2016 15:47:27 10.1.1.99 : %ASA-6-302014: Teardown TCP connection 3119818 for outside:209.98.44.193/49936 to inside:192.168.1.8/1547 duration 0:05:01 bytes 931 TCP Reset-I
Sep 7 15:47:28 10.1.1.99 Sep 07 2016 15:47:28 10.1.1.99 : %ASA-6-302020: Built outbound ICMP connection for faddr 192.168.151.96/512 gaddr 192.168.1.1/0 laddr 192.168.1.1/0

Re: Field extraction using regex -- Do I need to use an if or eval for this?

cpetterborg — Wed, 13 Dec 2017 19:54:08 GMT

Since you are talking Cisco ASA events, this seems to work quite well on the events that we collect:

rex "(?P<protocol>\w+)\s+connection"

In our data only 60 events out of 36,000,000 didn't match a protocol. The protocol wasn't always something caught correctly, however. Here is the breakdown of the value of protocol of the 36M events:

TCP - 69%
no - 11%
ICMP - 10%
UDP - 10%
matching - .05%

Everything else made up less than 200 events of the 36M. So if you eliminate no and matching from your results, you should get a pretty good result set:

rex "(?P<protocol>\w+)\s+connection" | search protocol=*
regex protocol!="(no|matching|closing|Dropped|deleting|allocate)"

There's plenty of opportunity to make changes to this, but it is at least a start.