I am using Splunk Enterprise on Windows machines and extract several fields from multiline events. Everything works fine as long the multiline events are smaller than about 600 chars. When the events are longer, some extracted fields are missing every time I try.
Example Event (a short one):
Application exception occurred:
Write a dump to file: D:\AVL\bin\..\log\GenGUI20180423_155507_5221_1229.dmp
App: D:\AVL\bin\GenGUI.exe P:5221 B:1229 Last called exception handler: D:\AVL\bin\GenGUI.exe
When: 2018-04-23;15:55:07.490
Reason: GenGUI.exe caused an EXCEPTION_ACCESS_VIOLATION in module at 0023:015C3CC1
Register: EAX=015C3CC0 EBX=015DC050 ECX=015C3CC0 EDX=0162AE60 ESI=00000000
EDI=00000000 EBP=04D2FA48 ESP=04D2FA18 EIP=015C3CC1 FLG=00010212
CS=0023 DS=002B SS=002B ES=002B FS=0053 GS=002B
Stacktrace:
My extractions:
Extraction AVL_Dump: (?m).*?to file: (?P(.*))[\r\n]App:
Extraction AVL_Prog: (?m).*?App: (?P(.*))P:
Extraction AVL_Exc_handle: (?m).*?exception handler: (?P(.*[\r\n]+)*)When:
Extraction AVL_Reason: (?m).*?Reason: (?P(.*[\r\n]+)*)Register:
My estimate is, that there is a parameter to set in props.conf, but which one ? I searched a while already and took a look at props docu, but did not find it.
Hello,
Please run below search to first check fields are extracted from all events (both short and long).
your_search...
| rex field=_raw "(?m).*dump\sto\sfile\:\s(?<AVL_DUMP>.+)[\r\n]App"
| rex field=_raw "(?m).*App\:\s(?<AVL_Prog>.+)P"
| rex field=_raw "(?m).*exception\shandler\:\s(?<AVL_Exec_handle>.+)[\r\n]?When"
| rex field=_raw "(?m).*Reason\:\s(?<AVL_Reason>.+)[\r\n]?Register"
If it is working, then use below extractions in your props.conf
EXTRACT-AVL_Dump = (?m).*dump\sto\sfile\:\s(?<AVL_Dump>.+)[\r\n]App
EXTRACT-AVL_Prog = (?m).*App\:\s(?<AVL_Prog>.+)P
EXTRACT-AVL_Exc_handle = (?m).*exception\shandler\:\s(?<AVL_Exec_handle>.+)[\r\n]?When
EXTRACT-AVL_Reason = (?m).*Reason\:\s(?<AVL_Reason>.+)[\r\n]?Register
Also for pasting special text into questions/comments, please use "Code Sample" option (101010 icon OR ctrl+k
shortcut)
Hello,
Please run below search to first check fields are extracted from all events (both short and long).
your_search...
| rex field=_raw "(?m).*dump\sto\sfile\:\s(?<AVL_DUMP>.+)[\r\n]App"
| rex field=_raw "(?m).*App\:\s(?<AVL_Prog>.+)P"
| rex field=_raw "(?m).*exception\shandler\:\s(?<AVL_Exec_handle>.+)[\r\n]?When"
| rex field=_raw "(?m).*Reason\:\s(?<AVL_Reason>.+)[\r\n]?Register"
If it is working, then use below extractions in your props.conf
EXTRACT-AVL_Dump = (?m).*dump\sto\sfile\:\s(?<AVL_Dump>.+)[\r\n]App
EXTRACT-AVL_Prog = (?m).*App\:\s(?<AVL_Prog>.+)P
EXTRACT-AVL_Exc_handle = (?m).*exception\shandler\:\s(?<AVL_Exec_handle>.+)[\r\n]?When
EXTRACT-AVL_Reason = (?m).*Reason\:\s(?<AVL_Reason>.+)[\r\n]?Register
Also for pasting special text into questions/comments, please use "Code Sample" option (101010 icon OR ctrl+k
shortcut)
It works for long events too ! Great !
Your extractions don't have names.. These will be located in props.conf
most likely on the search head. You can also go check under Settings > Fields
. What exactly are you trying to extract?
Extraction AVL_Dump: (?m).?to file: (?P(.))[\r\n]App:
Extraction AVL_Prog: (?m).?App: (?P(.))P:
Extraction AVL_Exc_handle: (?m).?exception handler: (?P(.[\r\n]+))When:
Extraction AVL_Reason: (?m).?Reason: (?P(.[\r\n]+))Register:
That's what I originally copied, but the variables names disappeard somehow when submitting.
The name is AVL_Dump in the first example, in '<>' . That works, I get the strings with the correct names - as long as the events are short.
So your were right, the Rex was bad defined. It was misleading that it worked for short ones...
It just happened again...I try to replace '<>' with '_'. Perhaps soemthings with German character sets...
Extraction AVL_Dump: (?m).*?to file: (?PAVL_Dump(.))[\r\n]App:
Extraction AVL_Prog: (?m).?App: (?PAVL_Prog(.))P:
Extraction AVL_Exc_handle: (?m).?exception handler: (?PAVL_Exc_handle(.[\r\n]+))When:
Extraction AVL_Reason: (?m).*?Reason: (?PAVL_Reason__(.[\r\n]+))Register:
Not yet exactly what I use in Splunk, but that extraction works for short events.
This means your regex is bad.. Can you show me exactly what your attempting to extract from the data? Perhaps post your data sample with the text you want extracted in bold
Now from WORD, but it dos not take the Bold. Anyway the extraction works fine for short events.
Application exception occurred:
Write a dump to file: D:\Avl\Bin..\Log\Gengui20180423_155507_5221_1229.Dmp
App: D:\AVL\bin\GenGUI.exe P:5221 B:1229 Last called exception handler: D:\AVL\bin\GenGUI.exe
When: 2018-04-23;15:55:07.490
Reason: GenGUI.exe caused an EXCEPTION_ACCESS_VIOLATION in module at 0023:015C3CC1
Register: EAX=015C3CC0 EBX=015DC050 ECX=015C3CC0 EDX=0162AE60 ESI=00000000
EDI=00000000 EBP=04D2FA48 ESP=04D2FA18 EIP=015C3CC1 FLG=00010212
CS=0023 DS=002B SS=002B ES=002B FS=0053 GS=002B
Stacktrace:
Extraction AVL_Dump: (?m).?to file: (?P(.))[\r\n]App:
Extraction AVL_Prog: (?m).?App: (?P(.))P:
Extraction AVL_Exc_handle: (?m).?exception handler: (?P(.[\r\n]+))When:
Extraction AVL_Reason: (?m).?Reason: (?P(.[\r\n]+))Register:
Here is a copy out of Splunk directly. The first short event shows all the extractions, the last longer only some of them. Always the same are missing.
06.07.18 11:36:53,823
Application exception occurred:
Write a dump to file: D:\AVL\bin..\log\CDEDeviceServer20180706_113653_7153_3508.dmp
App: D:\AVL\bin\CDEDeviceServer.exe P:7153 B:3508 Last called exception handler: D:\AVL\bin\CDEDeviceServer.exe
When: 2018-07-06;11:36:53.823
Reason: CDEDeviceServer.exe caused an EXCEPTION_ACCESS_VIOLATION in module CDEDeviceServer.exe at 0023:031A6F69
Register: EAX=00000000 EBX=0018FC94 ECX=00000001 EDX=0075D4B0 ESI=0018FC94
EDI=00000000 EBP=00000000 ESP=0018F6C0 EIP=031A6F69 FLG=00010246
CS=0023 DS=002B SS=002B ES=002B FS=0053 GS=002B
Stacktrace:
(null)
AVL_Dump = D:\AVL\bin..\log\CDEDeviceServer20180706_113653_7153_3508.dmp
AVL_Exc_handle = D:\AVL\bin\CDEDeviceServer.exe
AVL_Prog = D:\AVL\bin\CDEDeviceServer.exe
AVL_Pruefstand = 0685
AVL_Reason = CDEDeviceServer.exe caused an EXCEPTION_ACCESS_VIOLATION in module CDEDeviceServer.exe at 0023:031A6F69
›
06.07.18 11:08:03,408
Application exception occurred:
Write a dump to file: D:\AVL\bin..\log\CobraRuntimeAssembler20180706_110803_7153_3460.dmp
App: D:\AVL\bin\CobraRuntimeAssembler.exe P:7153 B:3460 Last called exception handler: D:\AVL\bin\CobraRuntimeAssembler.exe
When: 2018-07-06;11:08:03.408
Reason: CobraRuntimeAssembler.exe caused an in module KERNELBASE.dll at 0023:7534C54F, RaiseException()+0088 byte(s)
Register: EAX=09ADF434 EBX=E0434F4D ECX=00000001 EDX=00000000 ESI=09ADF4BC
EDI=003B32D0 EBP=09ADF484 ESP=09ADF434 EIP=7534C54F FLG=00000212
CS=0023 DS=002B SS=002B ES=002B FS=0053 GS=002B
Stacktrace:
0023:7534C54F KERNELBASE.dll, RaiseException()+0088 byte(s)
0023:66E47DA4 mscorwks.dll, GetCLRFunction()+6855 byte(s)
0023:66EA9570 mscorwks.dll, GetPrivateContextsPerfCounters()+184973 byte(s)
0023:66BFBBA0 mscorlib.ni.dll
0023:664D16D4 mscorlib.ni.dll
0023:66DD1E44 mscorwks.dll
0023:66EE8161 mscorwks.dll, IEE()+137377 byte(s)
0023:66EE81D0 mscorwks.dll, IEE()+137488 byte(s)
0023:670B0A67 mscorwks.dll, TranslateSecurityAttributes()+84436 byte(s)
0023:664E7306 mscorlib.ni.dll
0023:664F078F mscorlib.ni.dll
Alle 37 Zeilen anzeigen (=Show all 37 columns, my comment)
AVL_Dump = D:\AVL\bin..\log\CobraRuntimeAssembler20180706_110803_7153_3460.dmp
AVL_Prog = D:\AVL\bin\CobraRuntimeAssembler.exe
AVL_Pruefstand = 0680