topic Re: How do I write a regex for different field extractions according to event value? in Splunk Search

How do I write a regex for different field extractions according to event value?

eyirik — Fri, 02 Nov 2018 06:36:03 GMT

I have events like the ones below. I want to make a different field extraction according to the value of field MODEL.

MODEL values can be MD9EL, AVEV, PSJ . Each MODEL has different events and ı have to write a regex for the different models.

2018-11-02T06:24:16.000Z,MD9EL,NLTALLZNL01000005,36.99140,35.18779

2018-11-01T13:24:27.000Z,AVEV,NLTNVSZPL01000001,36.98865,35.19343,0.63,2512

2018-11-01T13:58:02.000Z,PSJ,NLTPNG23L01002172,,,,644,35,0,0,102,0,61,102

I think have to override the inputs according to the MODEL value. How can I process it?

Re: How do I write a regex for different field extractions according to event value?

FrankVl — Fri, 02 Nov 2018 10:51:49 GMT

I see two main options:

Use props and transforms to set specific sourcetypes based on the model. Then you can configure the required extractions for each of those specific sourcetypes. See: https://docs.splunk.com/Documentation/Splunk/latest/Data/Advancedsourcetypeoverrides

Use a generic sourcetype and write extractions that only match a certain model. Which would look something like this in props.conf:

EXTRACT-MD9EL = ^(?<timestamp>[^,]+),MD9EL,(?<fieldA>[^,]+),(?<fieldB>[^,]+)
EXTRACT-AVEV = ^(?<timestamp>[^,]+),AVEV,(?<fieldX>[^,]+),(?<fieldY>[^,]+),(?<fieldZ>[^,]+)... etc.

Re: How do I write a regex for different field extractions according to event value?

eyirik — Tue, 27 Nov 2018 12:55:08 GMT

Thanks. It works with EXTRACT-MD9EL. But this time , it extracts the model name and i cannot make model based search after field extraction. How can i add model name also as a field ?

Re: How do I write a regex for different field extractions according to event value?

FrankVl — Tue, 27 Nov 2018 13:55:12 GMT

Just put a capturing group around that part of the data as well:

EXTRACT-MD9EL = ^(?<timestamp>[^,]+),(?<modelname>MD9EL),(?<fieldA>[^,]+),(?<fieldB>[^,]+)
 EXTRACT-AVEV = ^(?<timestamp>[^,]+),(?<modelname>AVEV),(?<fieldX>[^,]+),(?<fieldY>[^,]+),(?<fieldZ>[^,]+)... etc.

Re: How do I write a regex for different field extractions according to event value?

eyirik — Tue, 11 Dec 2018 18:10:20 GMT

Hi. First regex is worked but second one is not worked.

EXTRACT-MD9EL = ^(?[^,]+),MD9EL,(?[^,]+),(?[^,]+)
EXTRACT-MD9EL = ^(?[^,]+),(?MD9EL),(?[^,]+),(?[^,]+)

when i want to show field model name , field extraction does not work.

Re: How do I write a regex for different field extractions according to event value?

macadminrohit — Tue, 11 Dec 2018 19:14:43 GMT

As per @FrankVl , you have to capture the named groups in the regex. Looks like you are not doing it.

And test your regex at www.regex101.com , its a great site.

Re: How do I write a regex for different field extractions according to event value?

eyirik — Tue, 11 Dec 2018 19:30:44 GMT

here is my sample event and my sample regex

event: 20181211T14:49:30.000Z,MD9EL,NLTALLZNL01000005,36.98827

regex working: EXTRACT-MD9EL = ^(?[^,]+),MD9EL,(?[^,]+),(?[^,]+)

regex not working: EXTRACT-MD9EL = ^(?[^,]+),(?MD9EL),(?[^,]+),(?[^,]+)

Where is my wrong ?

Re: How do I write a regex for different field extractions according to event value?

eyirik — Tue, 11 Dec 2018 20:04:59 GMT

I think i gt problem. In my event some values coming null. because of that extraction is not working. I send value 0, so regex worked.

Re: How do I write a regex for different field extractions according to event value?

FrankVl — Wed, 12 Dec 2018 09:05:54 GMT

That's good to hear! If my solution worked for you, please mark it as accepted, so it is clear for others that this question was successfully answered.

Re: How do I write a regex for different field extractions according to event value?

petom — Sun, 16 Dec 2018 18:14:33 GMT

@eyirik data sample you provided, seem to me similar for each MODEL. It's just that some fields are optional.
If it is so and you know what those fields mean, you can just use one universal regex and create a field extraction via UI or configure it in props.conf:

EXTRACT-models = ^(?<timestamp>.*?),(?<model>.*?),(?<fieldA>.*?)(?:,(?<fieldB>.*?))?(?:,(?<fieldC>.*?))?(?:,(?<fieldD>.*?))?(?:,(?<fieldE>.*?))?(?:,(?<fieldF>.*?))?(?:,(?<fieldG>.*?))?(?:,(?<fieldH>.*?))?(?:,(?<fieldI>.*?))?(?:,(?<fieldJ>.*?))?(?:,(?<fieldK>.*?))?(?:,(?<fieldL>.*?))?(?:,(?<extras>.*))?$

extras is there only for cases when there are some additional / unknown fields and those will be stored in the extras field.
And basically if you need to extend the number of fields to be extracted, just copy the sequence (?:,(?<fieldL>.*?))? over and over and change the name of the field.