Splunk Search

How do I write a regex for different field extractions according to event value?

eyirik
Explorer

I have events like the ones below. I want to make a different field extraction according to the value of field MODEL.

MODEL values can be MD9EL, AVEV, PSJ . Each MODEL has different events and ı have to write a regex for the different models.

2018-11-02T06:24:16.000Z,MD9EL,NLTALLZNL01000005,36.99140,35.18779

2018-11-01T13:24:27.000Z,AVEV,NLTNVSZPL01000001,36.98865,35.19343,0.63,2512

2018-11-01T13:58:02.000Z,PSJ,NLTPNG23L01002172,,,,644,35,0,0,102,0,61,102

I think have to override the inputs according to the MODEL value. How can I process it?

Tags (2)
0 Karma
1 Solution

FrankVl
Ultra Champion

I see two main options:

Use props and transforms to set specific sourcetypes based on the model. Then you can configure the required extractions for each of those specific sourcetypes. See: https://docs.splunk.com/Documentation/Splunk/latest/Data/Advancedsourcetypeoverrides

Use a generic sourcetype and write extractions that only match a certain model. Which would look something like this in props.conf:

EXTRACT-MD9EL = ^(?<timestamp>[^,]+),MD9EL,(?<fieldA>[^,]+),(?<fieldB>[^,]+)
EXTRACT-AVEV = ^(?<timestamp>[^,]+),AVEV,(?<fieldX>[^,]+),(?<fieldY>[^,]+),(?<fieldZ>[^,]+)... etc.

View solution in original post

0 Karma

petom
Path Finder

@eyirik data sample you provided, seem to me similar for each MODEL. It's just that some fields are optional.
If it is so and you know what those fields mean, you can just use one universal regex and create a field extraction via UI or configure it in props.conf:

EXTRACT-models = ^(?<timestamp>.*?),(?<model>.*?),(?<fieldA>.*?)(?:,(?<fieldB>.*?))?(?:,(?<fieldC>.*?))?(?:,(?<fieldD>.*?))?(?:,(?<fieldE>.*?))?(?:,(?<fieldF>.*?))?(?:,(?<fieldG>.*?))?(?:,(?<fieldH>.*?))?(?:,(?<fieldI>.*?))?(?:,(?<fieldJ>.*?))?(?:,(?<fieldK>.*?))?(?:,(?<fieldL>.*?))?(?:,(?<extras>.*))?$

extras is there only for cases when there are some additional / unknown fields and those will be stored in the extras field.
And basically if you need to extend the number of fields to be extracted, just copy the sequence (?:,(?<fieldL>.*?))? over and over and change the name of the field.

0 Karma

FrankVl
Ultra Champion

I see two main options:

Use props and transforms to set specific sourcetypes based on the model. Then you can configure the required extractions for each of those specific sourcetypes. See: https://docs.splunk.com/Documentation/Splunk/latest/Data/Advancedsourcetypeoverrides

Use a generic sourcetype and write extractions that only match a certain model. Which would look something like this in props.conf:

EXTRACT-MD9EL = ^(?<timestamp>[^,]+),MD9EL,(?<fieldA>[^,]+),(?<fieldB>[^,]+)
EXTRACT-AVEV = ^(?<timestamp>[^,]+),AVEV,(?<fieldX>[^,]+),(?<fieldY>[^,]+),(?<fieldZ>[^,]+)... etc.
0 Karma

eyirik
Explorer

Thanks. It works with EXTRACT-MD9EL. But this time , it extracts the model name and i cannot make model based search after field extraction. How can i add model name also as a field ?

0 Karma

FrankVl
Ultra Champion

Just put a capturing group around that part of the data as well:

EXTRACT-MD9EL = ^(?<timestamp>[^,]+),(?<modelname>MD9EL),(?<fieldA>[^,]+),(?<fieldB>[^,]+)
 EXTRACT-AVEV = ^(?<timestamp>[^,]+),(?<modelname>AVEV),(?<fieldX>[^,]+),(?<fieldY>[^,]+),(?<fieldZ>[^,]+)... etc.
0 Karma

eyirik
Explorer

Hi. First regex is worked but second one is not worked.

  1. EXTRACT-MD9EL = ^(?[^,]+),MD9EL,(?[^,]+),(?[^,]+)

  2. EXTRACT-MD9EL = ^(?[^,]+),(?MD9EL),(?[^,]+),(?[^,]+)

when i want to show field model name , field extraction does not work.

0 Karma

macadminrohit
Contributor

As per @FrankVl , you have to capture the named groups in the regex. Looks like you are not doing it.

And test your regex at www.regex101.com , its a great site.

0 Karma

eyirik
Explorer

here is my sample event and my sample regex

event: 20181211T14:49:30.000Z,MD9EL,NLTALLZNL01000005,36.98827

regex working: EXTRACT-MD9EL = ^(?[^,]+),MD9EL,(?[^,]+),(?[^,]+)

regex not working: EXTRACT-MD9EL = ^(?[^,]+),(?MD9EL),(?[^,]+),(?[^,]+)

Where is my wrong ?

0 Karma

eyirik
Explorer

I think i gt problem. In my event some values coming null. because of that extraction is not working. I send value 0, so regex worked.

0 Karma

FrankVl
Ultra Champion

That's good to hear! If my solution worked for you, please mark it as accepted, so it is clear for others that this question was successfully answered.

0 Karma
Get Updates on the Splunk Community!

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...

New Dates, New City: Save the Date for .conf25!

Wake up, babe! New .conf25 dates AND location just dropped!! That's right, this year, .conf25 is taking place ...

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...