Splunk Search

How to do fields extractions from multiline fields that have more than 600 characters?

sven_simon
New Member

I am using Splunk Enterprise on Windows machines and extract several fields from multiline events. Everything works fine as long the multiline events are smaller than about 600 chars. When the events are longer, some extracted fields are missing every time I try.
Example Event (a short one):
Application exception occurred:

Write a dump to file: D:\AVL\bin\..\log\GenGUI20180423_155507_5221_1229.dmp
App: D:\AVL\bin\GenGUI.exe P:5221 B:1229   Last called exception handler: D:\AVL\bin\GenGUI.exe
When: 2018-04-23;15:55:07.490
Reason: GenGUI.exe caused an EXCEPTION_ACCESS_VIOLATION in module  at 0023:015C3CC1
Register: EAX=015C3CC0  EBX=015DC050  ECX=015C3CC0  EDX=0162AE60  ESI=00000000
EDI=00000000  EBP=04D2FA48  ESP=04D2FA18  EIP=015C3CC1  FLG=00010212
CS=0023   DS=002B  SS=002B  ES=002B   FS=0053  GS=002B

Stacktrace:

My extractions:

Extraction AVL_Dump:   (?m).*?to file: (?P(.*))[\r\n]App:
Extraction AVL_Prog:   (?m).*?App: (?P(.*))P:
Extraction AVL_Exc_handle:  (?m).*?exception handler: (?P(.*[\r\n]+)*)When:
Extraction AVL_Reason:   (?m).*?Reason: (?P(.*[\r\n]+)*)Register:

My estimate is, that there is a parameter to set in props.conf, but which one ? I searched a while already and took a look at props docu, but did not find it.

0 Karma
1 Solution

sudosplunk
Motivator

Hello,

Please run below search to first check fields are extracted from all events (both short and long).

your_search...
| rex field=_raw "(?m).*dump\sto\sfile\:\s(?<AVL_DUMP>.+)[\r\n]App"
| rex field=_raw "(?m).*App\:\s(?<AVL_Prog>.+)P"
| rex field=_raw "(?m).*exception\shandler\:\s(?<AVL_Exec_handle>.+)[\r\n]?When"
| rex field=_raw "(?m).*Reason\:\s(?<AVL_Reason>.+)[\r\n]?Register"

If it is working, then use below extractions in your props.conf

EXTRACT-AVL_Dump = (?m).*dump\sto\sfile\:\s(?<AVL_Dump>.+)[\r\n]App
EXTRACT-AVL_Prog = (?m).*App\:\s(?<AVL_Prog>.+)P
EXTRACT-AVL_Exc_handle = (?m).*exception\shandler\:\s(?<AVL_Exec_handle>.+)[\r\n]?When
EXTRACT-AVL_Reason = (?m).*Reason\:\s(?<AVL_Reason>.+)[\r\n]?Register

Also for pasting special text into questions/comments, please use "Code Sample" option (101010 icon OR ctrl+k shortcut)

View solution in original post

sudosplunk
Motivator

Hello,

Please run below search to first check fields are extracted from all events (both short and long).

your_search...
| rex field=_raw "(?m).*dump\sto\sfile\:\s(?<AVL_DUMP>.+)[\r\n]App"
| rex field=_raw "(?m).*App\:\s(?<AVL_Prog>.+)P"
| rex field=_raw "(?m).*exception\shandler\:\s(?<AVL_Exec_handle>.+)[\r\n]?When"
| rex field=_raw "(?m).*Reason\:\s(?<AVL_Reason>.+)[\r\n]?Register"

If it is working, then use below extractions in your props.conf

EXTRACT-AVL_Dump = (?m).*dump\sto\sfile\:\s(?<AVL_Dump>.+)[\r\n]App
EXTRACT-AVL_Prog = (?m).*App\:\s(?<AVL_Prog>.+)P
EXTRACT-AVL_Exc_handle = (?m).*exception\shandler\:\s(?<AVL_Exec_handle>.+)[\r\n]?When
EXTRACT-AVL_Reason = (?m).*Reason\:\s(?<AVL_Reason>.+)[\r\n]?Register

Also for pasting special text into questions/comments, please use "Code Sample" option (101010 icon OR ctrl+k shortcut)

sven_simon
New Member

It works for long events too ! Great !

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Your extractions don't have names.. These will be located in props.conf most likely on the search head. You can also go check under Settings > Fields. What exactly are you trying to extract?

0 Karma

sven_simon
New Member

Extraction AVL_Dump: (?m).?to file: (?P(.))[\r\n]App:
Extraction AVL_Prog: (?m).?App: (?P(.))P:
Extraction AVL_Exc_handle: (?m).?exception handler: (?P(.[\r\n]+))When:
Extraction AVL_Reason: (?m).
?Reason: (?P(.[\r\n]+))Register:

That's what I originally copied, but the variables names disappeard somehow when submitting.
The name is AVL_Dump in the first example, in '<>' . That works, I get the strings with the correct names - as long as the events are short.

0 Karma

sven_simon
New Member

So your were right, the Rex was bad defined. It was misleading that it worked for short ones...

0 Karma

sven_simon
New Member

It just happened again...I try to replace '<>' with '_'. Perhaps soemthings with German character sets...
Extraction AVL_Dump: (?m).*?to file: (?P
AVL_Dump(.))[\r\n]App:
Extraction AVL_Prog: (?m).
?App: (?P
AVL_Prog(.))P:
Extraction AVL_Exc_handle: (?m).
?exception handler: (?P
AVL_Exc_handle(.[\r\n]+))When:
Extraction AVL_Reason: (?m).*?Reason: (?P
AVL_Reason__(.[\r\n]+))Register:

0 Karma

sven_simon
New Member

Not yet exactly what I use in Splunk, but that extraction works for short events.

0 Karma

skoelpin
SplunkTrust
SplunkTrust

This means your regex is bad.. Can you show me exactly what your attempting to extract from the data? Perhaps post your data sample with the text you want extracted in bold

0 Karma

sven_simon
New Member

Now from WORD, but it dos not take the Bold. Anyway the extraction works fine for short events.

Application exception occurred:
Write a dump to file: D:\Avl\Bin..\Log\Gengui20180423_155507_5221_1229.Dmp
App: D:\AVL\bin\GenGUI.exe P:5221 B:1229 Last called exception handler: D:\AVL\bin\GenGUI.exe
When: 2018-04-23;15:55:07.490
Reason: GenGUI.exe caused an EXCEPTION_ACCESS_VIOLATION in module at 0023:015C3CC1
Register: EAX=015C3CC0 EBX=015DC050 ECX=015C3CC0 EDX=0162AE60 ESI=00000000
EDI=00000000 EBP=04D2FA48 ESP=04D2FA18 EIP=015C3CC1 FLG=00010212
CS=0023 DS=002B SS=002B ES=002B FS=0053 GS=002B
Stacktrace:

Extraction AVL_Dump: (?m).?to file: (?P(.))[\r\n]App:
Extraction AVL_Prog: (?m).?App: (?P(.))P:
Extraction AVL_Exc_handle: (?m).?exception handler: (?P(.[\r\n]+))When:
Extraction AVL_Reason: (?m).
?Reason: (?P(.[\r\n]+))Register:

0 Karma

sven_simon
New Member

Here is a copy out of Splunk directly. The first short event shows all the extractions, the last longer only some of them. Always the same are missing.

06.07.18 11:36:53,823

Application exception occurred:
Write a dump to file: D:\AVL\bin..\log\CDEDeviceServer20180706_113653_7153_3508.dmp
App: D:\AVL\bin\CDEDeviceServer.exe P:7153 B:3508 Last called exception handler: D:\AVL\bin\CDEDeviceServer.exe
When: 2018-07-06;11:36:53.823
Reason: CDEDeviceServer.exe caused an EXCEPTION_ACCESS_VIOLATION in module CDEDeviceServer.exe at 0023:031A6F69
Register: EAX=00000000 EBX=0018FC94 ECX=00000001 EDX=0075D4B0 ESI=0018FC94
EDI=00000000 EBP=00000000 ESP=0018F6C0 EIP=031A6F69 FLG=00010246
CS=0023 DS=002B SS=002B ES=002B FS=0053 GS=002B
Stacktrace:
(null)

AVL_Dump = D:\AVL\bin..\log\CDEDeviceServer20180706_113653_7153_3508.dmp

AVL_Exc_handle = D:\AVL\bin\CDEDeviceServer.exe

AVL_Prog = D:\AVL\bin\CDEDeviceServer.exe

AVL_Pruefstand = 0685

AVL_Reason = CDEDeviceServer.exe caused an EXCEPTION_ACCESS_VIOLATION in module CDEDeviceServer.exe at 0023:031A6F69


06.07.18 11:08:03,408

Application exception occurred:
Write a dump to file: D:\AVL\bin..\log\CobraRuntimeAssembler20180706_110803_7153_3460.dmp
App: D:\AVL\bin\CobraRuntimeAssembler.exe P:7153 B:3460 Last called exception handler: D:\AVL\bin\CobraRuntimeAssembler.exe
When: 2018-07-06;11:08:03.408
Reason: CobraRuntimeAssembler.exe caused an in module KERNELBASE.dll at 0023:7534C54F, RaiseException()+0088 byte(s)
Register: EAX=09ADF434 EBX=E0434F4D ECX=00000001 EDX=00000000 ESI=09ADF4BC
EDI=003B32D0 EBP=09ADF484 ESP=09ADF434 EIP=7534C54F FLG=00000212
CS=0023 DS=002B SS=002B ES=002B FS=0053 GS=002B
Stacktrace:
0023:7534C54F KERNELBASE.dll, RaiseException()+0088 byte(s)
0023:66E47DA4 mscorwks.dll, GetCLRFunction()+6855 byte(s)
0023:66EA9570 mscorwks.dll, GetPrivateContextsPerfCounters()+184973 byte(s)
0023:66BFBBA0 mscorlib.ni.dll
0023:664D16D4 mscorlib.ni.dll
0023:66DD1E44 mscorwks.dll
0023:66EE8161 mscorwks.dll, IEE()+137377 byte(s)
0023:66EE81D0 mscorwks.dll, IEE()+137488 byte(s)
0023:670B0A67 mscorwks.dll, TranslateSecurityAttributes()+84436 byte(s)
0023:664E7306 mscorlib.ni.dll
0023:664F078F mscorlib.ni.dll

Alle 37 Zeilen anzeigen (=Show all 37 columns, my comment)

AVL_Dump = D:\AVL\bin..\log\CobraRuntimeAssembler20180706_110803_7153_3460.dmp

AVL_Prog = D:\AVL\bin\CobraRuntimeAssembler.exe

AVL_Pruefstand = 0680

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...