How do I write a regex for different field extractions according to event value?

[eyirik]

I have events like the ones below. I want to make a different field extraction according to the value of field MODEL.

MODEL values can be MD9EL, AVEV, PSJ . Each MODEL has different events and ı have to write a regex for the different models.

2018-11-02T06:24:16.000Z,MD9EL,NLTALLZNL01000005,36.99140,35.18779

2018-11-01T13:24:27.000Z,AVEV,NLTNVSZPL01000001,36.98865,35.19343,0.63,2512

2018-11-01T13:58:02.000Z,PSJ,NLTPNG23L01002172,,,,644,35,0,0,102,0,61,102

I think have to override the inputs according to the MODEL value. How can I process it?

[petom]

@eyirik data sample you provided, seem to me similar for each MODEL. It's just that some fields are optional.
If it is so and you know what those fields mean, you can just use one universal regex and create a field extraction via UI or configure it in props.conf:

EXTRACT-models = ^(?<timestamp>.*?),(?<model>.*?),(?<fieldA>.*?)(?:,(?<fieldB>.*?))?(?:,(?<fieldC>.*?))?(?:,(?<fieldD>.*?))?(?:,(?<fieldE>.*?))?(?:,(?<fieldF>.*?))?(?:,(?<fieldG>.*?))?(?:,(?<fieldH>.*?))?(?:,(?<fieldI>.*?))?(?:,(?<fieldJ>.*?))?(?:,(?<fieldK>.*?))?(?:,(?<fieldL>.*?))?(?:,(?<extras>.*))?$

extras is there only for cases when there are some additional / unknown fields and those will be stored in the extras field.
And basically if you need to extend the number of fields to be extracted, just copy the sequence (?:,(?<fieldL>.*?))? over and over and change the name of the field.

[FrankVl · Accepted solution]

I see two main options:

Use props and transforms to set specific sourcetypes based on the model. Then you can configure the required extractions for each of those specific sourcetypes. See: https://docs.splunk.com/Documentation/Splunk/latest/Data/Advancedsourcetypeoverrides

Use a generic sourcetype and write extractions that only match a certain model. Which would look something like this in props.conf:

EXTRACT-MD9EL = ^(?<timestamp>[^,]+),MD9EL,(?<fieldA>[^,]+),(?<fieldB>[^,]+)
EXTRACT-AVEV = ^(?<timestamp>[^,]+),AVEV,(?<fieldX>[^,]+),(?<fieldY>[^,]+),(?<fieldZ>[^,]+)... etc.

[eyirik]

Thanks. It works with EXTRACT-MD9EL. But this time , it extracts the model name and i cannot make model based search after field extraction. How can i add model name also as a field ?

[FrankVl]

Just put a capturing group around that part of the data as well:

EXTRACT-MD9EL = ^(?<timestamp>[^,]+),(?<modelname>MD9EL),(?<fieldA>[^,]+),(?<fieldB>[^,]+)
 EXTRACT-AVEV = ^(?<timestamp>[^,]+),(?<modelname>AVEV),(?<fieldX>[^,]+),(?<fieldY>[^,]+),(?<fieldZ>[^,]+)... etc.

[eyirik]

Hi. First regex is worked but second one is not worked.

1. EXTRACT-MD9EL = ^(?[^,]+),MD9EL,(?[^,]+),(?[^,]+)

2. EXTRACT-MD9EL = ^(?[^,]+),(?MD9EL),(?[^,]+),(?[^,]+)

when i want to show field model name , field extraction does not work.

[macadminrohit]

As per @FrankVl , you have to capture the named groups in the regex. Looks like you are not doing it.

And test your regex at www.regex101.com , its a great site.

[eyirik]

here is my sample event and my sample regex

event: 20181211T14:49:30.000Z,MD9EL,NLTALLZNL01000005,36.98827

regex working: EXTRACT-MD9EL = ^(?[^,]+),MD9EL,(?[^,]+),(?[^,]+)

regex not working: EXTRACT-MD9EL = ^(?[^,]+),(?MD9EL),(?[^,]+),(?[^,]+)

Where is my wrong ?

[eyirik]

I think i gt problem. In my event some values coming null. because of that extraction is not working. I send value 0, so regex worked.

[FrankVl]

That's good to hear! If my solution worked for you, please mark it as accepted, so it is clear for others that this question was successfully answered.