Use Regex to extract the fields

[siddharthmis]

I have data like-

2017-12-19 09:39:41|INFO|4b483c4b138de23b2f83a208c2313c4a|8de3f071aed6401d9ff5c4289694e852|a|b|c
2017-12-19 09:39:41|ERRO|4b483c4b138de23b2f83a208c2313c4a|8de3f071aed6401d9ff5c4289694e852|a|b|c|d

Since there are different number of fields, I am not able to extract these.

^(?<timestamp>[^|]+)\|(?<severity>[^|]+)\|(?<x>[^|]+)\|(?<y>[^|]+)\|(?<a>[^|]+)\|(?<b>[^|]+)\|(?<c>[^|^$]+)\|(?<d>.*) works for one but not the other.
Is there any way I can extract all fields.

[micahkemp]

Assuming your last values (a, b, c...) are of an unknown quantity, you could do something like this:

props.conf:

[<sourcetype>]
REPORT-sourcetypevalues = severityxy,othervalues

transforms.conf:

[severityxy]
REGEX = ^(?<timestamp>[^|]+)\|(?<severity>[^|]+)\|(?<x>[^|]+)\|(?<y>[^|]+)\|(?<othervalues>.*)$

[othervalues]
SOURCE_KEY = othervalues
REGEX = (^|\|)(?<othervalue>[^|]+)($|\|)
MV_ADD = true

[horsefez · Accepted solution]

Hey,

look at this:

https://regex101.com/r/reSnrj/1/
(if it doesn't load properly click in the field with the log events first, this will fix it)

^(?:(?<timestamp>[^|]+)\|(?<severity>[^|]+)\|(?<x>[^|]+)\|(?<y>[^|]+)\|(?<a>[^|]+)\|(?<b>[^|]+)\|(?<c>[^|]+)(?:\|(?<d>[^|]+))?)$

[cmerriman]

This isn't 100% there yet, still noodling, but it's something to start with:

^(?<timestamp>[^|]+)\|(?<severity>[^|]+)\|(?<x>[^|]+)\|(?<y>[^|]+)\|(?<a>[^|]+)\|(?<b>[^|]+)\|(?<c>[^|]+)?[\|]?(?<d>[^|].*)

this should extract them all, but it seems to but c inside d for the event that doesn't have d

[Elsurion]

you can use the same extraction but if the field d is optional you have to define it as optional

(\|(?<d>.*))?

then you can use this regex for both cases.

If you have more optional fields, then you can repeat the optional extraction.

[siddharthmis]

Can someone assist on this?

[micahkemp]

Is it always severity, x, y, then a variable number of pipe delimited values afterwards that should all be in one potentially multivalue) field?