How to extract data from log message data using rex field=_raw?

[sandysaahil]

Hi Gurus,

I am trying to extract data from log message using rex field=_raw. The regex I have is 

"Event <(?<eventNo>.*)>, Super <(?<super>.*)>, Charge <(?<oic>.*)>, number <(?<pcn>.*)>, Card <(?<cn>.*)>, CO <(?<co>.*)>, Warn <(?<warn>.*)>" | table _time oic eventNo pcn cn super co warn

and I am able to extract records. but the issue is may or may bot be present is the log and I still need to extract the rest of the data. I tried 

"Event <(?<eventNo>.*)>, Super <(?<super>.*)>, Charge <(?<oic>.*)>, number <(?<pcn>.*)>, Card <(?<cn>.*)>, (CO <(?<co>.*)>,)? Warn <(?<warn>.*)>" | table _time oic eventNo pcn cn super co warn

It gives me the records which does nto contain this item. I want to extract all the records irrestive of whether it is present or not present. Please let me know what am I doing wrong. Thanks a lot in advance.

[bowesmana]

Please provide examples of both types of data. I am assuming you are saying that CO may or may not be in the data...?

[sandysaahil]

You are right @bowesmana , the example data is 

Event <E523478>, Super <0>, Charge <N46965>, number <0199327578>, Card <2898005466>, Warn <true>
Event <E523478>, Super <0>, Charge <N46965>, number <0199327578>, Card <2898005466>, Warn <false>
Event <E521244>, Super <2>, Charge <N46965>, number <0199327750>, Card <2898005471>, CO <true>, Warn <true>

In third case, I get the value of CO which is missing from other two log statements. I want to ignore CO in other two but would like to extract CO if available.

[gcusello · Accepted solution]

Hi @sandysaahil,

in your logs you have a word thatr identifies each field, so you could create a regex for each field, in this way the other regexes aren't blocked when one field is missed, something like this:

index=your_index
| rex "Event \<(?<eventNo>\w*)"
| rex "Super \<(?<super>\d*)"
| rex "Charge \<(?<oic>\w*)"
| rex "number \<(?<pcn>\d*)"
| rex "Card \<(?<cn>\d*)"
| rex "CO \<(?<co>\w*)"
| rex "Warn \<(?<warn>\w*)" 
| table _time oic eventNo pcn cn super co warn

Ciao.

Giuseppe

[sandysaahil]

@gcusello This is working perfectly fine. Thanks for the solution. Just want to check one more thing. one of the rex fields contains hypens. I tried using rex "Expiry \<(?[\d-]*)>" . It onle fetches first record but ignores rest of the same expiry fields from other logs. I get blank column for that. Do you think I am doing something wrong here. It works perfectly for all other words which does not contain special characters.

[gcusello]

Hi @sandysaahil,

I hint to create a new question, so more people can help you better and quicker!

Anyway, in the sample you shared there isn't any hypen, could you share some sample with and without hypens?

Ciao.

Giuseppe

P.S.: Karma Points are appreciated by all the Contributors 😉