Field Extractions not working

jlixfeld — Thu, 08 May 2014 21:49:05 GMT

I'm trying to get field extractions to show up in the Interesting Fields.

My search string is as follows, and it completes successfully:

sourcetype=syslog rgw01.lab | rex field=_raw "%SEC-6-IPACCESSLOG.?.:  list (?P<log_acl_name>[A-Z]+\:[A-Z]+\:[A-Z]+) \w+ (?P<log_acl_proto>\w+) (?P<log_acl_sip>[\d]{1,3}\.[\d]{1,3}\.[\d]{1,3}\.[\d]{1,3})([(][0-9]+[)] |[ ])-> (?P<log_acl_dip>[\d]{1,3}\.[\d]{1,3}\.[\d]{1,3}\.[\d]{1,3})(?P<log_acl_dport>[(][0-9]+[)][,]|[,])"

I would expect these field extractions to show up, but they do not:

log_acl_name
log_acl_proto
log_acl_sip
log_acl_dip
log_acl_dport

I tried adding this to Settings > Fields > Field Extractions, but it still doesn't show up:

"%SEC-6-IPACCESSLOG.?.:  list (?P<log_acl_name>[A-Z]+\:[A-Z]+\:[A-Z]+) \w+ (?P<log_acl_proto>\w+) (?P<log_acl_sip>[\d]{1,3}\.[\d]{1,3}\.[\d]{1,3}\.[\d]{1,3})([(][0-9]+[)] |[ ])-> (?P<log_acl_dip>[\d]{1,3}\.[\d]{1,3}\.[\d]{1,3}\.[\d]{1,3})(?P<log_acl_dport>[(][0-9]+[)][,]|[,])"

I ran this regex through www.regexr.com and it matched everything I was interested in, so I used that as a template to construct the rex:

 %SEC-6-IPACCESSLOG.?.: list [A-Z]+\:[A-Z]+\:[A-Z]+ \w+ \w+ ([\d]{1,3}\.[\d]{1,3}\.[\d]{1,3}\.[\d]{1,3})([(][0-9]+[)] |[ ])-> ([\d]{1,3}\.[\d]{1,3}\.[\d]{1,3}\.[\d]{1,3})([(][0-9]+[)][,]|[,])

Here's some sample events:

2014-05-08T11:12:45.910030-04:00 lo21949.rgw01.lab.beanfield.com 193207: rgw01.lab: May  8 11:11:54.420: %SEC-6-IPACCESSLOGP: list FILTER:TV:IN permitted tcp 172.23.255.188(37548) -> 172.16.0.2(80), 1 packet
2014-05-08T11:12:35.937906-04:00 lo21949.rgw01.lab.beanfield.com 193206: rgw01.lab: May  8 11:11:44.448: %SEC-6-IPACCESSLOGRP: list FILTER:TV:IN permitted igmp 172.23.255.191 -> 224.0.0.22, 1 packet
2014-05-08T11:12:34.843132-04:00 lo21949.rgw01.lab.beanfield.com 193205: rgw01.lab: May  8 11:11:43.350: %SEC-6-IPACCESSLOGP: list FILTER:TV:IN permitted tcp 172.23.255.228(55094) -> 172.16.0.2(80), 1 packet
2014-05-08T11:12:33.806361-04:00 lo21949.rgw01.lab.beanfield.com 193204: rgw01.lab: May  8 11:11:42.316: %SEC-6-IPACCESSLOGRP: list FILTER:TV:IN permitted igmp 172.23.255.238 -> 224.0.0.22, 1 packet
2014-05-08T11:12:28.053939-04:00 lo21949.rgw01.lab.beanfield.com 193203: rgw01.lab: May  8 11:11:36.561: %SEC-6-IPACCESSLOGP: list FILTER:TV:IN permitted tcp 172.23.255.191(53347) -> 172.16.0.2(80), 1 packet
2014-05-08T11:12:07.076675-04:00 lo21949.rgw01.lab.beanfield.com 193201: rgw01.lab: May  8 11:11:15.584: %SEC-6-IPACCESSLOGP: list FILTER:TV:IN permitted tcp 172.23.255.223(58230) -> 172.16.0.2(80), 1 packet
2014-05-08T11:12:02.141604-04:00 lo21949.rgw01.lab.beanfield.com 193200: rgw01.lab: May  8 11:11:10.649: %SEC-6-IPACCESSLOGP: list FILTER:TV:IN permitted tcp 172.23.255.190(46180) -> 172.16.0.4(80), 1 packet
2014-05-08T11:11:53.415260-04:00 lo21949.rgw01.lab.beanfield.com 193199: rgw01.lab: May  8 11:11:01.922: %SEC-6-IPACCESSLOGP: list FILTER:TV:IN permitted tcp 172.23.255.238(35810) -> 172.16.0.2(80), 1 packet
2014-05-08T11:11:37.322462-04:00 lo21949.rgw01.lab.beanfield.com 193198: rgw01.lab: May  8 11:10:45.833: %SEC-6-IPACCESSLOGRP: list FILTER:TV:IN permitted igmp 172.23.255.249 -> 224.0.0.22, 1 packet
2014-05-08T11:11:36.274565-04:00 lo21949.rgw01.lab.beanfield.com 193197: rgw01.lab: May  8 11:10:44.784: %SEC-6-IPACCESSLOGRP: list FILTER:TV:IN permitted igmp 172.23.255.191 -> 224.0.0.22, 1 packet
2014-05-08T11:11:35.038938-04:00 lo21949.rgw01.lab.beanfield.com 193196: rgw01.lab: May  8 11:10:43.548: %SEC-6-IPACCESSLOGP: list FILTER:TV:IN permitted tcp 172.23.255.228(55093) -> 172.16.0.2(80), 1 packet
2014-05-08T11:11:32.034399-04:00 lo21949.rgw01.lab.beanfield.com 193195: rgw01.lab: May  8 11:10:40.544: %SEC-6-IPACCESSLOGRP: list FILTER:TV:IN permitted igmp 172.23.255.190 -> 224.0.0.22, 1 packet
2014-05-08T11:11:29.210428-04:00 lo21949.rgw01.lab.beanfield.com 193194: rgw01.lab: May  8 11:10:37.719: %SEC-6-IPACCESSLOGP: list FILTER:TV:IN permitted tcp 172.23.255.248(46516) -> 172.16.0.2(80), 1 packet
2014-05-08T11:11:21.422505-04:00 lo21949.rgw01.lab.beanfield.com 193193: rgw01.lab: May  8 11:10:29.929: %SEC-6-IPACCESSLOGP: list FILTER:TV:IN permitted tcp 172.23.255.249(59616) -> 172.16.0.2(80), 1 packet
2014-05-08T11:11:04.257287-04:00 lo21949.rgw01.lab.beanfield.com 193191: rgw01.lab: May  8 11:10:12.767: %SEC-6-IPACCESSLOGP: list FILTER:TV:IN permitted tcp 172.23.255.190(46178) -> 172.16.0.4(80), 1 packet
2014-05-08T11:10:53.425363-04:00 lo21949.rgw01.lab.beanfield.com 193190: rgw01.lab: May  8 11:10:01.935: %SEC-6-IPACCESSLOGP: list FILTER:TV:IN permitted tcp 172.23.255.238(35809) -> 172.16.0.2(80), 1 packet
2014-05-08T11:10:45.705140-04:00 lo21949.rgw01.lab.beanfield.com 193189: rgw01.lab: May  8 11:09:54.214: %SEC-6-IPACCESSLOGP: list FILTER:TV:IN permitted tcp 172.23.255.188(37544) -> 172.16.0.2(80), 1 packet
2014-05-08T11:10:36.785036-04:00 lo21949.rgw01.lab.beanfield.com 193188: rgw01.lab: May  8 11:09:45.295: %SEC-6-IPACCESSLOGRP: list FILTER:TV:IN permitted igmp 172.23.255.188 -> 224.0.0.22, 1 packet
2014-05-08T11:10:35.457510-04:00 lo21949.rgw01.lab.beanfield.com 193187: rgw01.lab: May  8 11:09:43.969: %SEC-6-IPACCESSLOGRP: list FILTER:TV:IN permitted igmp 172.23.255.191 -> 224.0.0.22, 1 packet
2014-05-08T11:10:32.421042-04:00 lo21949.rgw01.lab.beanfield.com 193185: rgw01.lab: May  8 11:09:40.929: %SEC-6-IPACCESSLOGRP: list FILTER:TV:IN permitted igmp 172.23.255.249 -> 224.0.0.22, 1 packet
2014-05-08T11:10:29.186012-04:00 lo21949.rgw01.lab.beanfield.com 193184: rgw01.lab: May  8 11:09:37.695: %SEC-6-IPACCESSLOGP: list FILTER:TV:IN permitted tcp 172.23.255.248(46514) -> 172.16.0.2(80), 1 packet
2014-05-08T11:10:28.054410-04:00 lo21949.rgw01.lab.beanfield.com 193183: rgw01.lab: May  8 11:09:36.561: %SEC-6-IPACCESSLOGP: list FILTER:TV:IN permitted tcp 172.23.255.191(53344) -> 172.16.0.2(80), 1 packet
2014-05-08T11:10:21.628942-04:00 lo21949.rgw01.lab.beanfield.com 193182: rgw01.lab: May  8 11:09:30.139: %SEC-6-IPACCESSLOGP: list FILTER:TV:IN permitted tcp 172.23.255.249(59613) -> 172.16.0.2(80), 1 packet
2014-05-08T11:10:16.498384-04:00 lo21949.rgw01.lab.beanfield.com 193181: rgw01.lab: May  8 11:09:25.008: %SEC-6-IPACCESSLOGP: list FILTER:TV:OUT permitted udp 172.16.5.4(50904) -> 232.16.2.17(2017), 600 packets
2014-05-08T11:10:16.498182-04:00 lo21949.rgw01.lab.beanfield.com 193180: rgw01.lab: May  8 11:09:25.008: %SEC-6-IPACCESSLOGP: list FILTER:TV:OUT permitted udp 172.16.5.15(34829) -> 232.16.2.160(2160), 600 packets
2014-05-08T11:10:06.881113-04:00 lo21949.rgw01.lab.beanfield.com 193178: rgw01.lab: May  8 11:09:15.390: %SEC-6-IPACCESSLOGP: list FILTER:TV:IN permitted tcp 172.23.255.223(58228) -> 172.16.0.2(80), 1 packet
2014-05-08T11:10:04.478482-04:00 lo21949.rgw01.lab.beanfield.com 193177: rgw01.lab: May  8 11:09:12.987: %SEC-6-IPACCESSLOGP: list FILTER:TV:IN permitted tcp 172.23.255.190(46177) -> 172.16.0.4(80), 1 packet
2014-05-08T11:09:51.405301-04:00 lo21949.rgw01.lab.beanfield.com 193176: rgw01.lab: May  8 11:08:59.914: %SEC-6-IPACCESSLOGP: list FILTER:TV:IN permitted tcp 172.23.255.249(59611) -> 172.16.0.2(80), 1 packet
2014-05-08T11:09:45.901948-04:00 lo21949.rgw01.lab.beanfield.com 193175: rgw01.lab: May  8 11:08:54.412: %SEC-6-IPACCESSLOGP: list FILTER:TV:IN permitted tcp 172.23.255.188(37543) -> 172.16.0.2(80), 1 packet

I'm not sure what else I might need to do... Any advice?

Thanks.

Re: Field Extractions not working

yannK — Thu, 08 May 2014 22:39:48 GMT

please provide a sample of the events to verify the rex.

Re: Field Extractions not working

jlixfeld — Fri, 09 May 2014 13:57:47 GMT

Thanks. Original post updated to include sample data.

Re: Field Extractions not working

somesoni2 — Fri, 09 May 2014 14:03:31 GMT

I see there is an extra space in your regex before word 'list'. I just removed it and your regex expression worked for your sample logs.

Re: Field Extractions not working

jlixfeld — Fri, 09 May 2014 14:21:29 GMT

Son of a ... Damn, how careless of me. Thanks, seems to work now.

topic Re: Field Extractions not working in Splunk Search

Field Extractions not working

Re: Field Extractions not working

Re: Field Extractions not working

Re: Field Extractions not working

Re: Field Extractions not working