Parsing of log using UF

[mukhan1]

Hello Team,

I was trying to parse my data by updating props.conf file i have created this file in (C:\Program Files\SplunkUniversalForwarder\etc\system\local\props.conf) 

[t24protocollog]
DATETIME_CONFIG=CURRENT
SHOULD_LINEMERGE=false
LINE_BREAKER=^@ID[\s\S]*?REMARK.*$
NO_BINARY_CHECK=true
disabled=false

by using above configuration file. 
When I'm using the regex it separate each line as a event instead of defining starting and endpoint of event. I checked this regex on regex 101 it is giving me proper result on regex101.com but not on Splunk. Attaching the screenshot how my logs are showing on SPLUNK GUI.

Below is the content of log file.
--------------------------------------

LIST F.PROTOCOL @ID PROTOCOL.ID PROCESS.DATE TIME.MSECS K.USER APPLICATION LEVEL.FUNCTION ID REMARK PAGE 1 11:34:02 23 NOV 2023

@ID............ 202403.16
@ID............ 202403.16
PROTOCOL.ID.... 202403.16
PROCESS.DATE... 20230926
TIME.MSECS..... 11:15:23:649
K.USER......... INPUTTER
APPLICATION.... DC.ENTRY
LEVEL.FUNCTION. 1
ID.............
REMARK......... QUIRY - AC.REPORT

@ID............ 202303.16
@ID............ 202303.16
PROTOCOL.ID.... 202303.16
PROCESS.DATE... 20230926
TIME.MSECS..... 11:15:23:649
K.USER......... INPUTTER
APPLICATION.... AC.ENTRY
LEVEL.FUNCTION. 1
ID.............
REMARK......... ENQUIRY - AC.REPORT

Kindly do let me know if im doing anything wrong, need to parse the log file.

[mukhan1]

Hello @gcusello ,

Thanks for your response, however I manually uploaded my sample logs the problem is when I'm providing the regex for event breaker it shows me result on each line making every line as event. I've uploaded the SS how it is showing me data you can see the attached file. 

Also you can take below sample logs 

LIST F.PROTOCOL @ID PROTOCOL.ID PROCESS.DATE TIME.MSECS K.USER APPLICATION LEVEL.FUNCTION ID REMARK PAGE 1 11:34:02 23 NOV 2023

@ID............ 202403.16
@ID............ 202403.16
PROTOCOL.ID.... 202403.16
PROCESS.DATE... 20230926
TIME.MSECS..... 11:15:23:649
K.USER......... INPUTTER
APPLICATION.... DC.ENTRY
LEVEL.FUNCTION. 1
ID.............
REMARK......... QUIRY - AC.REPORT

@ID............ 202303.16
@ID............ 202303.16
PROTOCOL.ID.... 202303.16
PROCESS.DATE... 20230926
TIME.MSECS..... 11:15:23:649
K.USER......... INPUTTER
APPLICATION.... AC.ENTRY
LEVEL.FUNCTION. 1
ID.............
REMARK......... ENQUIRY - AC.REPORT

by using this regex "^@ID[\s\S]*?REMARK.*$" i'm getting correct data on regex101. 

[gcusello]

Hi @mukhan1,

at first parsing is done on Indexers or (when present) on Heavy Forwarders, never on Universal Forwarders.

the only exception are INDEXED_EXTRACTIONS (as e.g. csv) where parsing starts on UF.

Anyway, see if it already exists an add-on for the technology you want to parse.

Then my hint is to take a sample of your logs and manually uoload it in your Splunk (using the Add Data feature) so you can use the guided sourcetype creation.

Ciao.

Giuseppe