Error with regex for syslog event

kseshadri — Tue, 03 Aug 2010 00:13:20 GMT

Running splunk on windows2003. I am getting the events but it seems my regex is not working right on the event.

Sample syslog event:

Feb 10 13:40:24 066.082.009.081 TPS,1,HN-IP,67.143.66.109,PC-IP,67.143.66.110,OUT,0,IN,0,update.intervideo.com:80,http://update.intervideo.com/Ping.asp

should be converted to the following fields

FORMAT = host::$1 record_type::$2 hn_ip::$3 pc_ip::$4 bytes_out::$5 bytes_in::$6 domain::$7

However it is not working.. could someone give me a clue as to whats wrong with the regex ?

REGEX = ^.*[^\:]*\d\d\:\d\d\:\d\d[^\:]*?\s(\d+\.\d+\.\d+\.\d+)\s[^,]+,([^,]+),HN-IP,([^,]+),PC-IP,([^,]+),OUT,([^,]+),IN,([^,]+),([^,]+),[[bc_uri]]

[Hughes_TPS]
#Feb 10 13:40:24 066.082.009.081 TPS,1,HN-IP,67.143.66.109,PC-IP,67.143.66.110,OUT,0,IN,0,update.intervideo.com:80,http://update.intervideo.com/Ping.asp
REGEX = ^.*[^\:]*\d\d\:\d\d\:\d\d[^\:]*?\s(\d+\.\d+\.\d+\.\d+)\s[^,]+,([^,]+),HN-IP,([^,]+),PC-IP,([^,]+),OUT,([^,]+),IN,([^,]+),([^,]+),[[bc_uri]]
FORMAT = host::$1 record_type::$2 hn_ip::$3 pc_ip::$4 bytes_out::$5 bytes_in::$6 domain::$7

inputs.conf

[udp://514]
sourcetype = syslog
no_appending_timestamp = true

[host::192.168.10:81]
sourcetype = HughesTPS
KV_MODE = none
REPORT-syslog = Hughes_TPS
SEGMENTATION = inner

Re: Error with regex for syslog event

Lowell — Tue, 03 Aug 2010 04:52:08 GMT

I'm not sure you can mix-and match numbered group-based extraction with splunk's advanced regex (e.g. [[bc_url]])

Perhaps this would work if you used named groups. You could try using a tranforms.props like this:

BTW, you don't want to override the "host" field like this. If you want to explicitly set the "host" value that get's indexed, you should do it with a special index-time transformer.

[Hughes_TPS_host]
#Feb 10 13:40:24 066.082.009.081
DEST_KEY = MetaData:Host
REGEX    = ^.*[^\:]*\d\d\:\d\d\:\d\d[^\:]*?\s(\d+\.\d+\.\d+\.\d+)\s
FORMAT   = host::$1

[Hughes_TPS]
#Feb 10 13:40:24 066.082.009.081 TPS,1,HN-IP,67.143.66.109,PC-IP,67.143.66.110,OUT,0,IN,0,update.intervideo.com:80,http://update.intervideo.com/Ping.asp
REGEX = ^.*[^\:]*\d\d\:\d\d\:\d\d[^\:]*?\s\d+\.\d+\.\d+\.\d+\s[^,]+,(?P<record_type>[^,]+),HN-IP,(?P<hn_ip>[^,]+),PC-IP,(?P<pc_ip>[^,]+),OUT,(?P<bytes_out>[^,]+),IN,(?P<bytes_in>[^,]+),(?P<domain>[^,]+),(?P<url>.+)

If you get the entry above to work, then try using the fancy combo splunk regex thing....

[Hughes_TPS]
#Feb 10 13:40:24 066.082.009.081 TPS,1,HN-IP,67.143.66.109,PC-IP,67.143.66.110,OUT,0,IN,0,update.intervideo.com:80,http://update.intervideo.com/Ping.asp
REGEX = ^.*[^\:]*\d\d\:\d\d\:\d\d[^\:]*?\s\d+\.\d+\.\d+\.\d+\s[^,]+,(?P<record_type>[^,]+),HN-IP,(?P<hn_ip>[^,]+),PC-IP,(?P<pc_ip>[^,]+),OUT,(?P<bytes_out>[^,]+),IN,(?P<bytes_in>[^,]+),(?P<domain>[^,]+),[[bc_uri]]

Your props.conf should probably look something like this:

[host::192.168.10]
sourcetype = HughesTPS

[HughesTPS]
KV_MODE = none
TRANSFORM-host = Hughes_TPS_host
REPORT-syslog = Hughes_TPS
# I wouldn't recommend messing with segmentation unless you're and advanced user.
# SEGMENTATION = inner

topic Re: Error with regex for syslog event in Splunk Search

Error with regex for syslog event

Re: Error with regex for syslog event