topic Re: Unable to extract optional fields in splunk through regex in All Apps and Add-ons

Unable to extract optional fields in splunk through regex

luv — Fri, 15 Nov 2013 15:38:52 GMT

Hi, I have logs some what like this

......[ABC] - [YUP1,AConsole]
......[PQR] - [YUP1,PConsole]
......[ZAD] - [YUP1,DConsole]
......[SID] - [TYU3,2013-08-29,QConsole]
......[POP] - [TYU3,2013-08-30,TConsole]
......[IOL] - [TYU3,2013-09-01,XConsole]
......[DSW] - [GKFO,2013-09-12,iConsole,Payment1]
......[ESD] - [IOSD,2013-09-13,iConsole,Payment2]
......[ABC] - [YUP1,AConsole]
......[RTS] - [YUP1,RConsole]
......[SID] - [TYU3,2013-09-26,QConsole]
......[DSW] - [GKFO,2013-10-29,iConsole,Payment3]
......[EDS] - [EDC1,FConsole]

In the square brackets [(Field1),(Field2),(Field3),(Field4)] some of the fields are optional, they come and go in some events.

1st combination [(Field1),(Field3)]
2nd combination [(Field1),(Field2),(Field3)]
3rd combination [(Field1),(Field2),(Field3),(Field4)]
This means Field1 & Field3 are always there in the events but Field2 & Field4 are optional, I want to extract all of them.

This is the regex which i have come up with, But seems like it is not working for my case

rex field=_raw "\] - \[(?< Field1 >[^,]+)(,)?(?< Field2 >[^,]+)?,(?< Field3 >[^,\]]+)(,)?(< Field4 >[^\]]+)?"
PS- I have added a space between "<" and ">" because it text box was escaping it.
Any help would be really appreciated

Thanks 🙂

Re: Unable to extract optional fields in splunk through regex

dmaislin_splunk — Fri, 15 Nov 2013 15:46:54 GMT

You said: This means Field1 & Field3 are always there in the events but Field3 & Field4 are optional, I want to extract all of them.

Did you mean Field2 & Field4 are optional?

Re: Unable to extract optional fields in splunk through regex

somesoni2 — Fri, 15 Nov 2013 16:01:44 GMT

You can use following. Assumption there are atleast 2 fields and they are separated by comma. You can add more eval statements if you have more fields.

..your search..| rex field=_raw ".*\[(?<fieldlist>[^\]]+)" |eval fieldlist=split(fieldlist,",") 
| eval fieldcount=mvcount(fieldlist) 
| eval field1=mvindex(fieldlist,0) 
| eval field2=case(fieldcount > 2,mvindex(fieldlist,1),1=1,"")
| eval field3=case(fieldcount > 2,mvindex(fieldlist,2),1=1,mvindex(fieldlist,1))
| eval field4=case(fieldcount > 3,mvindex(fieldlist,3),1=1,"")

Re: Unable to extract optional fields in splunk through regex

dmaislin_splunk — Fri, 15 Nov 2013 16:20:32 GMT

I think this will also work:

.+\[(?<FIELD1>[^,]+)\,?(?<FIELD2>[0-9-]+)?\,(?<FIELD3>\w+)\,?(?<FIELD4>\S+)?\]

Re: Unable to extract optional fields in splunk through regex

luv — Fri, 15 Nov 2013 19:52:23 GMT

that worked!!! thank you so much 🙂

Re: Unable to extract optional fields in splunk through regex

luv — Fri, 15 Nov 2013 19:54:00 GMT

This is really cool way to extract fields,it worked....Will definitely try more of this in future 🙂

Re: Unable to extract optional fields in splunk through regex

dmaislin_splunk — Fri, 15 Nov 2013 19:59:40 GMT

Please accept this answer by checking the check mark. Thanks!

Re: Unable to extract optional fields in splunk through regex

dmaislin_splunk — Fri, 15 Nov 2013 20:22:21 GMT

Thanks much!

Re: Unable to extract optional fields in splunk through regex

luv — Fri, 15 Nov 2013 20:47:34 GMT

just out of curiosity whats this "\" for?? between ".+\[(?[^,]+)" and ","
same with "?(?[0-9-]+)?" and ","
is there something which needs to be escaped?

Re: Unable to extract optional fields in splunk through regex

dmaislin_splunk — Fri, 15 Nov 2013 20:56:24 GMT

You probably don't need to escape the comma but it never hurts to be literal. I should have just escaped the last one too.

Re: Unable to extract optional fields in splunk through regex

d29priyanka — Tue, 07 Apr 2015 19:52:13 GMT

Hi I have events look like this:

DISKBSIZE,T0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
DISKBSIZE,T3729,8.0,4.0,0.0,7.4,4.0,4.0,4.0,0.0,10.8,0.0,0.0,6.0,0.0,4.0,6.0,0.0,10.5,0.0,8.0,4.0,4.0,0.0,8.7,4.0,8.4,5.7,4.0,9.3,0.0,5.5,0.0
DISKBSIZE,T3728,10.2,4.0,0.0,0.0,5.7,27.8,5.7,8.6
DISKBSIZE,T3729,0.0,4.0,11.3,0.0,0.0,10.8,0.0,6.0,0.0,5.3,4.0,0.0,11.1,0.0,4.0,6.0,0.0,5.6,0.0,13.9,0.0,4.0,5.3,17.1,0.0,9.3,0.0,10.0,5.4,6.7,4.0,13.2,0.0,8.0,0.0

Some fields come and go.Need a regex which extracts multiple fields.
Here is the regex which i came up with

|rex field=_raw "(?i)DISKBSIZE,(?P[^,]),(?P[^,]\d+),(?P[^,]\d+),(?P[^,]\d+),(?P[^,]\d+),(?P[^,]\d+),(?P[^,]\d+),(?P[^,]\d+),(?P[^,]\d+),(?P[^,]\d+),?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?"