topic Re: Field Extaction from regex not displaying Character in Splunk Search

Field Extaction from regex not displaying Character

pmcfadden91 — Thu, 16 Jul 2015 22:03:31 GMT

Hi, I have an issue for extracting values. It extracts 7 out of the 8 characters I need to catch.
I currently have this:

 index=gfs_etd_mft  source="/opt/TMWD/SecureTransport/var/logs/xferlog" | rex field=_raw "(?i)\..*? (?P<FIELDNAME>**[i|n|p|o|r|a|j])**" | dedup FIELDNAME | table FIELDNAME

It does return all the characters listed in the regex onto the statistics table. However, when I add the character "b"( rex field=_raw "(?i)\..*? (?P<FIELDNAME>**[i|n|b|p|o|r|a|j]**)"), it only displays "a" and "b" in stats table. How do I correct?

Re: Field Extaction from regex not displaying Character

MuS — Thu, 16 Jul 2015 22:32:24 GMT

can you provide some log samples?

Re: Field Extaction from regex not displaying Character

woodcock — Thu, 16 Jul 2015 23:24:42 GMT

It looks like your character class is inefficient and you are not escaping your asterisks; try this:

index=gfs_etd_mft  source="/opt/TMWD/SecureTransport/var/logs/xferlog" | rex field=_raw "(?i)\..*? (?<FIELDNAME>\*\*[inbporaj])\*\*" | dedup FIELDNAME | table FIELDNAME

Re: Field Extaction from regex not displaying Character

landen99 — Fri, 17 Jul 2015 12:45:55 GMT

Your regex indicates that there may be a pattern like ".stuff **a**" We really need to see the raw data and the desired result before we can understand the best regex for it. I am thinking that the following may extract you results much better:

index=gfs_etd_mft  source="/opt/TMWD/SecureTransport/var/logs/xferlog" | rex field=_raw "(?i)\*{2}(?<FIELDNAME>[inbporaj])\*{2}" | dedup FIELDNAME | table FIELDNAME

Re: Field Extaction from regex not displaying Character

pmcfadden91 — Mon, 20 Jul 2015 14:38:33 GMT

Below are some log examples:

Fri Jul 17 14:22:15 2015 0 139.149.36.161 10032 /sbclocal/InternalSecureFileTransfer/users/ETDIT/gmiprod2skctest/chi/global_idt/comm/idts_00104.zip **b s i r** gmiprod2skctest ssh 0 *

Fri Jul 17 11:28:10 2015 0 localhost 0 /sbclocal/InternalSecureFileTransfer/users/ETDIT/gmi_test_ftp/gmi_to_skc/gmi/SKCU02/asia_memo/account_types.zip **b n p r** gmi_test_ftp folder 0 *

Fri Jul 17 11:03:29 2015 8 151.191.80.226 3341520 /sbclocal/InternalSecureFileTransfer/users/ETDIT/tpt_ftp_test/outbox/rexuat/ldn/GMI_OBS_IRSCME_POS_20150716.txt **a n i r** tpt_ftp_test ftp 0 *

Fri Jul 17 11:03:29 2015 1 rex_ldn_uat1_cmp.ldn.swissbank.com 3341520 /sbclocal/InternalSecureFileTransfer/users/ETDIT/tpt_ftp_test/outbox/rexuat/ldn/GMI_OBS_IRSCME_POS_20150716.txt **a s o r** tpt_ftp_test ssh 0 *

It allows me to extract the combinations as individual characters into a field until I add I add the character "b". Also I was wondering if there is a way to extract different combinations(as shown in examples) and the single characters of the combos all into one field? I have a regex that pulls all combinations now. I just wanted to combine them.

 index=gfs_etd_mft  source="/opt/TMWD/SecureTransport/var/logs/xferlog" | rex field=_raw "(?i)\..*? (?P<FIELDNAME>\w+\s+\w+\s+\w+\s+\w+)\s+\w+" | dedup FIELDNAME | table FIELDNAME

Re: Field Extaction from regex not displaying Character

landen99 — Mon, 20 Jul 2015 15:28:56 GMT

On your sample data, the following regex extracts the letters:

\*{2}(?P<FIELDNAME>[^\*]+)\*{2}

MATCH 1
FIELDNAME [150-157] b s i r
MATCH 2
FIELDNAME [339-346] b n p r
MATCH 3
FIELDNAME [539-546] a n i r
MATCH 4
FIELDNAME [756-763] a s o r

' It allows me to extract the combinations as individual characters into a field until I add I add the character "b" '

If I understood correctly about the part of matching everything until you get to a "b", the regex becomes:

\*{2}(?P<FIELDNAME>[^b\*]+)\*{2}

MATCH 1
FIELDNAME [539-546] a n i r
MATCH 2
FIELDNAME [756-763] a s o r

You can then use makemv command to create a multivalue field of each string of single characters.

| rex "\*{2}(?P<FIELDNAME>[^b\*]+)\*{2}"
| makemv FIELDNAME delim=" " allowempty=t

Re: Field Extaction from regex not displaying Character

pmcfadden91 — Mon, 20 Jul 2015 16:23:01 GMT

This worked when I plugged it in and tweaked a bit. Thanks!

Re: Field Extaction from regex not displaying Character

landen99 — Mon, 20 Jul 2015 17:01:50 GMT

No problem. I used regex101.com to craft the regex. Try it out if you like.