Extract multivalue field using transforms mv_add=true not working as expected

[ak9092]

Hi,

I am having hard time extracting multi value fields present in an event using transforms mv_add=true, it seems to be partially working by just extracting the first and third value present in the event but skipping the second and the fourth value.

The regex which i am using seems to be perfectly matching for all the values in regex101 but not sure why Splunk is unable to capture it all.

Following is the sample event and regex I am using -

Event -

postreport=test_west_policy\;passed\;(first_post:status:passed:pass_condition[clear]:fail_condition[]:skip_condition[]\;second_post:status:skipped:pass_condition[clear]:fail_condition[]:skip_condition[timed_out]\;third_post:status:failed:pass_condition[]:fail_condition[error]:skip_condition[]\;fourth_post:status:passed:pass_condition[clear]:fail_condition[]:skip_condition[])

Regex - https://regex101.com/r/r66eOz/1 

(?<=\(|]\\;)(?<post>[^:]+):status:(?<status>[^:]*):pass_condition\[(?<passed_condition>[^\]]*)\]:fail_condition\[(?<failed_condition>[^\]]*)\]:skip_condition\[(?<skipped_condition>[^\]]*)\]

so splunk is just matching all values for first_post and third_post in above event and skipping the second_post & fourth_post..

the same regex i tried with rex command and in that it just matches first_post field values  -

|rex field=raw_msg max_match=0 "(?<=\(|]\\;)(?<post>[^:]+):status:(?<status>[^:]*):pass_condition\[(?<passed_condition>[^\]]*)\]:fail_condition\[(?<failed_condition>[^\]]*)\]:skip_condition\[(?<skipped_condition>[^\]]*)\]"

Can someone please help me figure if i am missing something here.

Thanks.

[ITWhisperer]

With Splunk rex you need to double up on backslashes when matching backslashes in the string - try something like this

| rex field=raw_msg max_match=0 "(?<=\(|]\\\\;)(?<post>[^:]+):status:(?<status>[^:]*):pass_condition\[(?<passed_condition>[^\]]*)\]:fail_condition\[(?<failed_condition>[^\]]*)\]:skip_condition\[(?<skipped_condition>[^\]]*)\]"

Having said that, you might want to consider extracting each group of fields as a whole and use mvexpand before separating into post, status, etc. as the multivalue fields you currently have do not align as the null values are not inserted into the mv fields

[ak9092]

Thanks @ITWhisperer , the additional backslash seems to be doing the trick for rex command but still no luck having this worked with transforms.conf mv_add=true setting. Basically i need this fields to be available at search time hence trying to figure out a way for that.  And when you say extract each group of fields as a whole what you mean by that. Can you please help me with an example to better understand that approach ?

[ITWhisperer]

| rex field=raw_msg max_match=0 "(?<=\(|]\\\\;)(?<group>[^:]+:status:[^:]*:pass_condition\[[^\]]*\]:fail_condition\[[^\]]*\]:skip_condition\[[^\]]*)\]"

[ak9092]

@ITWhisperer , this will be good if am doing transforming search using mvexpand but any idea on how i can achieve the same results through search time fields extractions using props & transforms.conf

[bluelobster]

Hi @ak9092 , did you find any solution for this?