Splunk Search

How do I do a field extraction with a special character?

Explorer

Hello, I have some logs that required to extract the fields. the raw data is in the format as below.

"xxx","yyy","zzz","sd323ds","3343tds23@343"

I would like to extract the field like this
field 1: xxx
field 2: yyy
field 3: zzz
field 4: sd323ds
field 5: 3343tds23@343

I use the field extractor, and it almost did what I want. The regular expression extracts the field by the double quotes, but there are some cases where double quotes may appear in some field contents, such as

"xxx","yyy","zzz","sd323ds","3343"Hello World" tds23@343"

I want the field 5 is 3343"Hello World" tds23@343, but it cannot be done by the current regular expression.

How can I modify my existing regular expression to achieve this?

0 Karma
1 Solution

Motivator

Hi!
Your logs look very much like a CSV file. How about you add a CSV header to your log file:

field_1,field_2,field_3,field_4,field_5
"xxx6","yyy","zzz","sd323ds","3343tds23@343"
"xxx","yyy","zzz","sd323ds","3343"Hello World" tds23@343"
...

Then you only need to set sourcetype=csv on your input. Splunk will extract the fields automatically.

Alternatively, have a look at this regular expression:

^"(?<field_1>.*)","(?<field_2>.*)","(?<field_3>.*)","(?<field_4>.*)","(?<field_5>.*)"$

Example on regex101.com

View solution in original post

Champion

Try this

| makeresults 
| eval test= "\"xxx\",\"yyy\",\"zzz\",\"sd323ds\",\"3343\"Hello World\" tds23@343\"" 
| rex field=test "\"(?P<field1>.*)\",\"(?P<field2>.*)\",\"(?P<field3>.*)\",\"(?P<field4>.*)\",\"(?<field5>.*)"
0 Karma

Motivator

Hi!
Your logs look very much like a CSV file. How about you add a CSV header to your log file:

field_1,field_2,field_3,field_4,field_5
"xxx6","yyy","zzz","sd323ds","3343tds23@343"
"xxx","yyy","zzz","sd323ds","3343"Hello World" tds23@343"
...

Then you only need to set sourcetype=csv on your input. Splunk will extract the fields automatically.

Alternatively, have a look at this regular expression:

^"(?<field_1>.*)","(?<field_2>.*)","(?<field_3>.*)","(?<field_4>.*)","(?<field_5>.*)"$

Example on regex101.com

View solution in original post

Explorer

Thank you very much for your suggestion which almost perfect for my need. I modified a little bit such that the line break is also accepted. For my understanding the expression is separating the fields by the "," . I am asked that if the content is containing this pattern, fields will be extracted wrongly. Is it possible for regular expression to handle such the case that may be something like "escape character"? I am not good at regular expression and have no idea on this.

0 Karma
Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes and swag!