Extracting specific parts from _raw logs

[DanAlexander]

Hi All,

Can anyone help me create a regex to extract the bolded parts from the following _raw log, please?

meta sequenceId="182311942"]10000 - [action:"Accept"; ........; origin:"10.111.10.111"; originsicname:"CN=................610;policy_name=High_Trust-1\]"; dst:"192.168.11.01"; log_delay:"1683724684"; layer_name:"Some text"; layer_name:"High_Trust-1 Application"; layer_uuid:"426c8a................."StoneBeat-Control"; src:"192.168.81.62"]

Thank you in advance!

[yuanliu]

It seems that your developers take pains to design a well formatted log.  It would be a waste to use regex for extraction.  Use extract instead.

| extract pairdelim=";" kvdelim=":"

Hope this helps 

[DanAlexander]

Thanks for the update @yuanliu 

Would you please elaborate on the regex waste? 

Not so sure what you have in mind based on your experience.

[yuanliu]

To extract patterns like ":\s*(?<field1>[^;]+)[^:]+(?<field2>[^;]+)" (which is required for that type of data), rex has to scan character by character with an indeterministic presumption.  In comparison, pairdelim=";" kvdelim=":" simply scans for fixed strings ";" and ":", which is computationally less complex. (And less demanding in RAM.)  As @isoutamo said, this does not mean that extract will always be more efficient or any choice will have material impact on performance.  But as a general practice, choose fixed pattern over regex.  The main advantage, of course, is that extract command extracts multiple kv pairs regardless of their order.

[isoutamo]

Just replace those all rex statements with this one. This will extract all those kv pairs.

Which option is more efficient can be check by Job Inspector.

[cklunck]

After the extraction is complete, are you hoping to have fields and field values like the following?

action=Accept

origin=10.111.10.111

layer_name="Some text"

[DanAlexander]

Thanks for the reply @cklunck 

Positive, this is what I want to achieve.

[isoutamo · Accepted solution]

Hi

if above is true you could try this for all previous fields as own rex. When you’re using separate rex, then the order can change and you could add more fields later.

<your base search>
| rex "action:\"(<?action>[^\"]+)"
| <next rex with another field name> …

r. Ismo