topic Re: Regex Extraction on multiple lines in Splunk Search

How to do regex Extraction on multiple lines?

chidiuchegbu — Mon, 25 Apr 2022 15:27:01 GMT

I have been avoiding RegEx for quite sometime in Splunk but I now I really need to deal with it and understand it.

I really need help with this three cases;

First Case:

I have events that usually start with things like 57A,53A and followed by other strings. I want to match 57A but the field value will be FMDKNTLA

:57A:3232324646
FMDKNTLA

Second Case:

Another extraction example will be extracting field 31A but the value will be "NKN"

:32A:200117NKN200000000,00

Third Case:

extracting field 31A but the value will be "200000000,00"

:31A:200117NKN200000000,00

Any help will be appreciated

Re: Regex Extraction on multiple lines

isoutamo — Mon, 25 Apr 2022 10:35:27 GMT

here is some examples, maybe not the perfect ones as I haven't so enough samples what you have.

| makeresults | fields _time | eval _raw = ":57A:3232324646 FMDKNTLA" | rex "^:(?<field>[^:]+)"

2) I'm exception that you are meaning 32A not 31A? In that case option 1 works also in this case.

3) Also in this case option 1 should work.

https://regex101.com is nice place to work & test those.

r. Ismo

Re: Regex Extraction on multiple lines

chidiuchegbu — Mon, 25 Apr 2022 11:10:54 GMT

Thanks for the feedback. Yes I have been using regex101.com to learn.

Throwing more light to this. This is a multiple lines of strings of data.The only static field is the 57A,32A but the others are dynamic and changes with data

For use case 2 and 3

Only 32A is static field but the "NKN" and the digits before and after it changes with each data

32A:200117NKN200000000,00

for use case 1 below, only 57A is constant, "FMDKNTLA" is dynamic and can be "ASTGNMLA" with next data

:57A:3232324646
FMDKNTLA

So I want have it extracted as

use case 1 - field and field value : >>>57A - FMDKNTLA but can be "57A - ASTGNMLA" next time

use case 2 - field and field value: >>> 32A - NKN but can be "32A - USD" next time

use case 3 - field and field value: >>> 32A - 200000000,00 but can be "32A - 3400000" next time

Dont know if this helps

Re: Regex Extraction on multiple lines

ITWhisperer — Mon, 25 Apr 2022 11:48:52 GMT

| makeresults | eval _raw=":57A:3232324646 FMDKNTLA :32A:200117NKN200000000,00 :32A:200117NKN200000000,00" | multikv noheader=t | table _raw | eval _raw=replace(_raw," "," ") ``` the lines above set up data as posted ``` | rex ":57A:\d+\s*(?<case1>\w+)" | rex ":32A:\d+(?<case2>[[:upper:]]+)(?<case3>[\d\,]+)"

Re: Regex Extraction on multiple lines

chidiuchegbu — Mon, 25 Apr 2022 12:20:10 GMT

I will try it out

Re: Regex Extraction on multiple lines

chidiuchegbu — Mon, 25 Apr 2022 12:47:05 GMT

Hi ITWhisperer,

Thanks a lot. Case 2 and case 3 works but case 1 is not matching. Could you please review again? note that

:57A:3232324646 FMDKNTLA are not on the same line in the logs but like the below

:57A:3232324646

FMDKNTLA

Re: Regex Extraction on multiple lines

chidiuchegbu — Mon, 25 Apr 2022 12:52:41 GMT

Note that the number in between ":3232324646 " for case 1 :57A:3232324646 FMDKNTLA should be skipped

only

:57A - FMDKNTLA needed as field and value respectively but again 57A is on a different line from FMDKNTLA represented in the raw log like the below

57A:3232324646
FMDKNTLA

Re: Regex Extraction on multiple lines

ITWhisperer — Mon, 25 Apr 2022 12:54:57 GMT

That's why I put the replace in when setting up the data to change the space to a new line.

I am assuming the data is still in one event?

Re: Regex Extraction on multiple lines

ITWhisperer — Mon, 25 Apr 2022 12:58:38 GMT

Re: Regex Extraction on multiple lines

chidiuchegbu — Mon, 25 Apr 2022 15:57:18 GMT

Thank you. I will try it out again. Though I do not understand fully how it all come together. Its a good place to begin my learning.

Yes the data is on one event with other data --serveral lines , upto 20 lines but the below is the focus of this extraction, while the others will come separately since am not sure if all can in one rex..

.................................................

:53A:/D/3232324646
FMDKNTLA
:57A:/D/3123424646
FTDMNSLA

so I will extract 53A data separately matching to dynamic data "FMDKNTLA" and same with 57A as you have rightly done

Re: Regex Extraction on multiple lines

chidiuchegbu — Mon, 25 Apr 2022 15:59:32 GMT

If you can put little explanation to each syntax that will really help

Re: Regex Extraction on multiple lines

ITWhisperer — Mon, 25 Apr 2022 17:07:27 GMT

OK where did the /D/ come from? That wasn't in your original description of the events - this is why the extract doesn't match for the first case

Re: Regex Extraction on multiple lines

chidiuchegbu — Mon, 25 Apr 2022 17:28:04 GMT

Oka it must have been wrongly copied from regex101 as I have been testing this there.

However, this content is entirely dynamic - meaning it can change to anything. So my focus

:57A:
FIDTNGLA

Re: Regex Extraction on multiple lines

ITWhisperer — Mon, 25 Apr 2022 17:42:15 GMT

rex works by matching strings to find anchors so that strings can be extracted. This relies on there being a pattern. You have now given yet another version of this field. Please can you give more examples (a definitive list?) and explain which part you want extracting.

Re: Regex Extraction on multiple lines

chidiuchegbu — Mon, 25 Apr 2022 18:59:10 GMT

I did not give another version. am just saying that the certain fields are not static while others are. at the end of the day what I want to extract is

:57A:
FIDTNGLA

This is a sample event which I posted earlier...This is one event with many strings

:53A:/D/3232324646
FMDKNTLA
:57A:/D/3123424646
FTDMNSLA

So I need 53A as field and FMDKNTLA as the value

and separately needs 57A as the field and FTDMNSLA.

Also there are other lines preceding the events above (same event) but I have successfully extracted them

hope this is clear

Re: Regex Extraction on multiple lines

ITWhisperer — Mon, 25 Apr 2022 21:56:59 GMT

| rex ":57A:\S*(?<case1a>\w+)" | rex ":53A:\S*(?<case1b>\w+)"

Re: Regex Extraction on multiple lines

chidiuchegbu — Tue, 26 Apr 2022 09:00:13 GMT

This is still not working and am not sure why. I have just modified sensitive info to paste the full event below.

${1:A29FDDTMLAXXX4444204444}{6:{107:200971602}{441:0}}{1:A29FDDTMLAXXX4444204444}{2:O1092343420017IKNITMLAAXXX45559999999001161692N}{3:{105:NGM}{141:0070}{111:c7837321-c49e-4ccc-b555-05492e60dfd5}{115:1230000}}{6:
:20:KKNXXX33333333
:23B:DREM
:23E:KDVA
:26T:001
:32A:200617NKN115003980,00
:51K:/2222222222
CAMA TECHNOLOGY COMPANY LIM
MITED
:53A:/D/0000024849
FMDKNTLA
:57A:/D/0000024946
FTDMNSLA
:59:/4440555666
MITRALO COMPANY
:71A:OWW
:72:/COMTYPTN/021
/BNN/CAMA TECHNOLOGY
-}{5:{LAC:00000000}{LAC:00000000}{KMK:0000000000}}{S:{KAC:}{DAC:}{WOP:S}}