What's the best way to extract key value pairs from the following log?

[Raghav2384]

Hello Experts,

I am trying to extract key-value pairs from the following. Here's the sample log. I have tried using xmlkv, spath and no luck.

"2014-11-11 04:46:13|xyz|INFO|#ae5760fa|Sweepstakes|RESPONSE=REDEEMED_ON11-11-2014,04:46:06BATCH_TYPE2CUST_FIRST_NAMEjacobBATCH_EXPIRATION11-12-2014,04:43:16CUST_EMAIL_CONFIRMEDNVOUCHER_IDZBEB2RLKQZOBCUST_INTERNET_PROVIDEROtherPURCHASE_COMPLETEYVOUCHER_COUNT1CUST_LASt_NAMEGomathamDEVICE_LIMIT0POLICY_TYPE1CUST_MAC_ADDRESSabc1.ab12.a123.1234CREATED_BYcustCREATED_ON11-11-2014,04:43:16BATCH_IDCUST_REGISTERED_ON11-11-2014,04:43:16CUST_EMAILxyz@yahoo.comBATCH_STATUSA","2014-11-11T04:46:13.000-0500"

What's the easiest way to extract first and second as $field1::$field2 or a on the fly search time extraction?

Appreciate your help.

Thanks,
Rgv

[musskopf · Accepted solution]

So, based on that here how you could start with:

index=bla "your search" | rex field=_raw "FIRST_NAME(?P<fieldA>\w+)BATCH_EXPIRATION.+LASt_NAME(?P<fieldB>\w+)DEVICE_LIMIT" | table fieldA, fieldB

This is just an example assuming all your events have a similar format, you should be able to extract other fields as your requirement. Again, just "past" your log into http://regex101.com and start with the Regex on my example.

[musskopf]

Just two more things... you probably notice you could choose any name to the "fieldA" (B, etc). Also looking to your data, should be straight forward to expand this extraction method for almost all fields.

Once you have it done, you also could configure it inside "Settings -> Fields -> Field Extraction" so you don't need to do it every search using the rex command.

Cheers

[musskopf]

Hey Raghav, you definitely need to use regular expressions... Before I suggest anything, is the text you paste correct? looks like is missing "whitespaces" and the line breaking is wrong... Try to paste the text and use the "code sample" icon on the top of the form to format it correctly, like this:

test 123

As regular expression suggestion, I always recommend to use the site http://regex101.com/ to play with.

[Raghav2384]

Unfortunately, that's the format they are in. Multilines with no spaces 😞

[musskopf]

Ok, so just clarify, what you mean by extract "first and second as $field1::$field2"? What are the parts of the text you wish to extract as fields, pls give me an example, like:

fieldA=jacob
fieldB=Gomatham
fieldC=???
etc

Cheers

[Raghav2384]

Yup...you got it