topic Re: Extracting specific parts from _raw logs in Splunk Search

Extracting specific parts from _raw logs

DanAlexander — Wed, 10 May 2023 16:39:48 GMT

Hi All,

Can anyone help me create a regex to extract the bolded parts from the following _raw log, please?

meta sequenceId="182311942"]10000 - [action:"Accept"; ........; origin:"10.111.10.111"; originsicname:"CN=................610;policy_name=High_Trust-1\]"; dst:"192.168.11.01"; log_delay:"1683724684"; layer_name:"Some text"; layer_name:"High_Trust-1 Application"; layer_uuid:"426c8a................."StoneBeat-Control"; src:"192.168.81.62"]

Thank you in advance!

Re: Extracting specific parts from _raw logs

cklunck — Thu, 27 Jul 2023 04:43:26 GMT

After the extraction is complete, are you hoping to have fields and field values like the following?

action=Accept

origin=10.111.10.111

layer_name="Some text"

Re: Extracting specific parts from _raw logs

isoutamo — Thu, 27 Jul 2023 05:04:34 GMT

if above is true you could try this for all previous fields as own rex. When you’re using separate rex, then the order can change and you could add more fields later.

<your base search> | rex "action:\"(<?action>[^\"]+)" | <next rex with another field name> …

r. Ismo

Re: Extracting specific parts from _raw logs

yuanliu — Thu, 27 Jul 2023 05:16:20 GMT

It seems that your developers take pains to design a well formatted log. It would be a waste to use regex for extraction. Use extract instead.

| extract pairdelim=";" kvdelim=":"

Hope this helps

Re: Extracting specific parts from _raw logs

DanAlexander — Thu, 27 Jul 2023 08:15:58 GMT

Thanks for the reply @cklunck

Positive, this is what I want to achieve.

Re: Extracting specific parts from _raw logs

DanAlexander — Thu, 27 Jul 2023 08:17:54 GMT

Thanks for the update @yuanliu

Would you please elaborate on the regex waste?

Not so sure what you have in mind based on your experience.

Re: Extracting specific parts from _raw logs

isoutamo — Thu, 27 Jul 2023 10:00:42 GMT

Just replace those all rex statements with this one. This will extract all those kv pairs.

Which option is more efficient can be check by Job Inspector.

Re: Extracting specific parts from _raw logs

yuanliu — Thu, 27 Jul 2023 16:42:55 GMT

To extract patterns like ":\s*(?<field1>[^;]+)[^:]+(?<field2>[^;]+)" (which is required for that type of data), rex has to scan character by character with an indeterministic presumption. In comparison, pairdelim=";" kvdelim=":" simply scans for fixed strings ";" and ":", which is computationally less complex. (And less demanding in RAM.) As @isoutamo said, this does not mean that extract will always be more efficient or any choice will have material impact on performance. But as a general practice, choose fixed pattern over regex. The main advantage, of course, is that extract command extracts multiple kv pairs regardless of their order.