topic Re: How to do fields extractions from multiline fields that have more than 600 characters? in Splunk Search

How to do fields extractions from multiline fields that have more than 600 characters?

sven_simon — Mon, 30 Jul 2018 11:49:25 GMT

I am using Splunk Enterprise on Windows machines and extract several fields from multiline events. Everything works fine as long the multiline events are smaller than about 600 chars. When the events are longer, some extracted fields are missing every time I try.
Example Event (a short one):
Application exception occurred:

Write a dump to file: D:\AVL\bin\..\log\GenGUI20180423_155507_5221_1229.dmp
App: D:\AVL\bin\GenGUI.exe P:5221 B:1229   Last called exception handler: D:\AVL\bin\GenGUI.exe
When: 2018-04-23;15:55:07.490
Reason: GenGUI.exe caused an EXCEPTION_ACCESS_VIOLATION in module  at 0023:015C3CC1
Register: EAX=015C3CC0  EBX=015DC050  ECX=015C3CC0  EDX=0162AE60  ESI=00000000
EDI=00000000  EBP=04D2FA48  ESP=04D2FA18  EIP=015C3CC1  FLG=00010212
CS=0023   DS=002B  SS=002B  ES=002B   FS=0053  GS=002B

Stacktrace:

My extractions:

Extraction AVL_Dump:   (?m).*?to file: (?P(.*))[\r\n]App:
Extraction AVL_Prog:   (?m).*?App: (?P(.*))P:
Extraction AVL_Exc_handle:  (?m).*?exception handler: (?P(.*[\r\n]+)*)When:
Extraction AVL_Reason:   (?m).*?Reason: (?P(.*[\r\n]+)*)Register:

My estimate is, that there is a parameter to set in props.conf, but which one ? I searched a while already and took a look at props docu, but did not find it.

Re: How to do fields extractions from multiline fields that have more than 600 characters?

skoelpin — Mon, 30 Jul 2018 13:30:20 GMT

Your extractions don't have names.. These will be located in props.conf most likely on the search head. You can also go check under Settings > Fields. What exactly are you trying to extract?

Re: How to do fields extractions from multiline fields that have more than 600 characters?

sven_simon — Tue, 29 Sep 2020 20:40:43 GMT

Extraction AVL_Dump: (?m).?to file: (?P(.))[\r\n]App:
Extraction AVL_Prog: (?m).?App: (?P(.))P:
Extraction AVL_Exc_handle: (?m).?exception handler: (?P(.[\r\n]+))When:
Extraction AVL_Reason: (?m).?Reason: (?P(.[\r\n]+))Register:

That's what I originally copied, but the variables names disappeard somehow when submitting.
The name is AVL_Dump in the first example, in '<>' . That works, I get the strings with the correct names - as long as the events are short.

Re: How to do fields extractions from multiline fields that have more than 600 characters?

sven_simon — Tue, 29 Sep 2020 20:40:45 GMT

It just happened again...I try to replace '<>' with '_'. Perhaps soemthings with German character sets...
Extraction AVL_Dump: (?m).*?to file: (?PAVL_Dump(.))[\r\n]App:
Extraction AVL_Prog: (?m).?App: (?PAVL_Prog(.))P:
Extraction AVL_Exc_handle: (?m).?exception handler: (?PAVL_Exc_handle(.[\r\n]+))When:
Extraction AVL_Reason: (?m).*?Reason: (?PAVL_Reason__(.[\r\n]+))Register:

Re: How to do fields extractions from multiline fields that have more than 600 characters?

sven_simon — Mon, 30 Jul 2018 13:43:34 GMT

Not yet exactly what I use in Splunk, but that extraction works for short events.

Re: How to do fields extractions from multiline fields that have more than 600 characters?

skoelpin — Mon, 30 Jul 2018 13:47:20 GMT

This means your regex is bad.. Can you show me exactly what your attempting to extract from the data? Perhaps post your data sample with the text you want extracted in bold

Re: How to do fields extractions from multiline fields that have more than 600 characters?

sven_simon — Tue, 29 Sep 2020 20:40:48 GMT

Now from WORD, but it dos not take the Bold. Anyway the extraction works fine for short events.

Application exception occurred:
Write a dump to file: D:\Avl\Bin..\Log\Gengui20180423_155507_5221_1229.Dmp
App: D:\AVL\bin\GenGUI.exe P:5221 B:1229 Last called exception handler: D:\AVL\bin\GenGUI.exe
When: 2018-04-23;15:55:07.490
Reason: GenGUI.exe caused an EXCEPTION_ACCESS_VIOLATION in module at 0023:015C3CC1
Register: EAX=015C3CC0 EBX=015DC050 ECX=015C3CC0 EDX=0162AE60 ESI=00000000
EDI=00000000 EBP=04D2FA48 ESP=04D2FA18 EIP=015C3CC1 FLG=00010212
CS=0023 DS=002B SS=002B ES=002B FS=0053 GS=002B
Stacktrace:

Re: How to do fields extractions from multiline fields that have more than 600 characters?

sven_simon — Tue, 29 Sep 2020 20:40:51 GMT

Here is a copy out of Splunk directly. The first short event shows all the extractions, the last longer only some of them. Always the same are missing.

06.07.18 11:36:53,823

Application exception occurred:
Write a dump to file: D:\AVL\bin..\log\CDEDeviceServer20180706_113653_7153_3508.dmp
App: D:\AVL\bin\CDEDeviceServer.exe P:7153 B:3508 Last called exception handler: D:\AVL\bin\CDEDeviceServer.exe
When: 2018-07-06;11:36:53.823
Reason: CDEDeviceServer.exe caused an EXCEPTION_ACCESS_VIOLATION in module CDEDeviceServer.exe at 0023:031A6F69
Register: EAX=00000000 EBX=0018FC94 ECX=00000001 EDX=0075D4B0 ESI=0018FC94
EDI=00000000 EBP=00000000 ESP=0018F6C0 EIP=031A6F69 FLG=00010246
CS=0023 DS=002B SS=002B ES=002B FS=0053 GS=002B
Stacktrace:
(null)

AVL_Dump = D:\AVL\bin..\log\CDEDeviceServer20180706_113653_7153_3508.dmp

AVL_Exc_handle = D:\AVL\bin\CDEDeviceServer.exe

AVL_Prog = D:\AVL\bin\CDEDeviceServer.exe

AVL_Pruefstand = 0685

AVL_Reason = CDEDeviceServer.exe caused an EXCEPTION_ACCESS_VIOLATION in module CDEDeviceServer.exe at 0023:031A6F69

›
06.07.18 11:08:03,408

Application exception occurred:
Write a dump to file: D:\AVL\bin..\log\CobraRuntimeAssembler20180706_110803_7153_3460.dmp
App: D:\AVL\bin\CobraRuntimeAssembler.exe P:7153 B:3460 Last called exception handler: D:\AVL\bin\CobraRuntimeAssembler.exe
When: 2018-07-06;11:08:03.408
Reason: CobraRuntimeAssembler.exe caused an in module KERNELBASE.dll at 0023:7534C54F, RaiseException()+0088 byte(s)
Register: EAX=09ADF434 EBX=E0434F4D ECX=00000001 EDX=00000000 ESI=09ADF4BC
EDI=003B32D0 EBP=09ADF484 ESP=09ADF434 EIP=7534C54F FLG=00000212
CS=0023 DS=002B SS=002B ES=002B FS=0053 GS=002B
Stacktrace:
0023:7534C54F KERNELBASE.dll, RaiseException()+0088 byte(s)
0023:66E47DA4 mscorwks.dll, GetCLRFunction()+6855 byte(s)
0023:66EA9570 mscorwks.dll, GetPrivateContextsPerfCounters()+184973 byte(s)
0023:66BFBBA0 mscorlib.ni.dll
0023:664D16D4 mscorlib.ni.dll
0023:66DD1E44 mscorwks.dll
0023:66EE8161 mscorwks.dll, IEE()+137377 byte(s)
0023:66EE81D0 mscorwks.dll, IEE()+137488 byte(s)
0023:670B0A67 mscorwks.dll, TranslateSecurityAttributes()+84436 byte(s)
0023:664E7306 mscorlib.ni.dll
0023:664F078F mscorlib.ni.dll

Alle 37 Zeilen anzeigen (=Show all 37 columns, my comment)

AVL_Dump = D:\AVL\bin..\log\CobraRuntimeAssembler20180706_110803_7153_3460.dmp

AVL_Prog = D:\AVL\bin\CobraRuntimeAssembler.exe

AVL_Pruefstand = 0680

Re: How to do fields extractions from multiline fields that have more than 600 characters?

sudosplunk — Mon, 30 Jul 2018 15:31:45 GMT

Hello,

Please run below search to first check fields are extracted from all events (both short and long).

your_search...
| rex field=_raw "(?m).*dump\sto\sfile\:\s(?<AVL_DUMP>.+)[\r\n]App"
| rex field=_raw "(?m).*App\:\s(?<AVL_Prog>.+)P"
| rex field=_raw "(?m).*exception\shandler\:\s(?<AVL_Exec_handle>.+)[\r\n]?When"
| rex field=_raw "(?m).*Reason\:\s(?<AVL_Reason>.+)[\r\n]?Register"

If it is working, then use below extractions in your props.conf

EXTRACT-AVL_Dump = (?m).*dump\sto\sfile\:\s(?<AVL_Dump>.+)[\r\n]App
EXTRACT-AVL_Prog = (?m).*App\:\s(?<AVL_Prog>.+)P
EXTRACT-AVL_Exc_handle = (?m).*exception\shandler\:\s(?<AVL_Exec_handle>.+)[\r\n]?When
EXTRACT-AVL_Reason = (?m).*Reason\:\s(?<AVL_Reason>.+)[\r\n]?Register

Also for pasting special text into questions/comments, please use "Code Sample" option (101010 icon OR ctrl+k shortcut)

Re: How to do fields extractions from multiline fields that have more than 600 characters?

sven_simon — Tue, 31 Jul 2018 09:24:54 GMT

It works for long events too ! Great !

Re: How to do fields extractions from multiline fields that have more than 600 characters?

sven_simon — Tue, 31 Jul 2018 10:58:12 GMT

So your were right, the Rex was bad defined. It was misleading that it worked for short ones...