Regex of .txt file in Splunk

[mukhan1]

Hello,

I'm integrating the .txt file in Splunk, however while integrating the file my events are breaking into single line not all events but many of them are breaking into single line. Attaching the log file in comments.

Below is how my data is appearing on Splunk when I add this txt file into Splunk. Is there any way I can limit the starting and ending point of my event. I want my data to be started from @ID and ends on REMARK. 

 
And if I use regex "(@ID[\s\S]*?REMARK[\s\S]*?)(?=@ID|$)" while adding the data, many of my logs are getting missing attaching the snapshot of it also. not sure how to resolve this issue,  if anyone can know how i can integrate this .txt file to get my event start from (@ID to REMARK)

[PickleRick]

Generally, you should avoid using SHOULD_LINEMERGE=true whenever you can.

In your case it seems  like something like this (along with SHOULD_LINEMERGE=false) should work

LINE_BREAKER = ^REMARK[^\r\n]+([\r\n]+)@ID

[mukhan1]

Thanks for the response, when I'm using this line breaker regex  alot of events are missing attaching screenshot as reference.
Is there any way i can read that whole file into single payload, If yes this can resolve my issue. 

[PickleRick]

That is kinda strange.

If you check it on regex101 - https://regex101.com/r/Bavlui/1 (I have no idea how long the saved regexes are kept) - it seems to work. As you can see, the group 1 is properly matched to the space between events.

So there might be something not 100% copy-pasteable and your events might actually look a bit different (maybe have some hanging spaces/tabs or something like that).

In general, your LINE_BREAKER should match the place on which you want to break the stream into separate events and must contain a capturing group which will match the part which separates one event from another. That group will be discarded as the "spacer" between events.

[mukhan1]

LIST F.PROTOCOL @ID PROTOCOL.ID PROCESS.DATE TIME.MSECS K.USER APPLICATION LEVEL.FUNCTION ID REMARK PAGE 1 11:34:02 23 NOV 2023

@ID............ 202309260081340532.21
@ID............ 202309260081340532.21
PROTOCOL.ID.... 202309260081340532.21
PROCESS.DATE... 20230926
TIME.MSECS..... 11:15:32:934
K.USER......... INPUTTER
APPLICATION.... AC.INWARD.ENTRY
LEVEL.FUNCTION. 1
ID.............
REMARK......... ENQUIRY - AC.INTERFACE.REPORT

@ID............ 202309260081340523.16
@ID............ 202309260081340523.16
PROTOCOL.ID.... 202309260081340523.16
PROCESS.DATE... 20230926
TIME.MSECS..... 11:15:23:649
K.USER......... INPUTTER
APPLICATION.... AC.INWARD.ENTRY
LEVEL.FUNCTION. 1
ID.............
REMARK......... ENQUIRY - AC.INTERFACE.REPORT

@ID............ 202309260081340465.12
@ID............ 202309260081340465.12
PROTOCOL.ID.... 202309260081340465.12
PROCESS.DATE... 20230926
TIME.MSECS..... 11:14:25:781
K.USER......... INPUTTER
APPLICATION.... AC.INWARD.ENTRY
LEVEL.FUNCTION. 1
ID.............
REMARK......... ENQUIRY - AC.INTERFACE.REPORT

@ID............ AUTHORISER-8232
@ID............ AUTHORISER-8232
PROTOCOL.ID.... AUTHORISER-8232
PROCESS.DATE... 20230926
TIME.MSECS..... 09:08:19:962
K.USER......... AUTHORISER
APPLICATION.... PGM.BREAK
LEVEL.FUNCTION. 1
ID............. LIST F.PROTOCOL @ID PROTOCOL.ID PROCESS.DATE TIME.MSECS K.USER APPLICATION LEVEL.FUNCTION ID REMARK PAGE 2 11:34:02 23 NOV 2023

REMARK.........

@ID............ 202309260081340530.06
@ID............ 202309260081340530.06
PROTOCOL.ID.... 202309260081340530.06
PROCESS.DATE... 20230926
TIME.MSECS..... 11:15:30:223
K.USER......... INPUTTER
APPLICATION.... AC.INWARD.ENTRY
LEVEL.FUNCTION. 1
ID.............
REMARK......... ENQUIRY - AC.INTERFACE.REPORT

@ID............ 202309269535047401.01
@ID............ 202309269535047401.01
PROTOCOL.ID.... 202309269535047401.01
PROCESS.DATE... 20230926
TIME.MSECS..... 13:10:01:201
K.USER......... INPUTTER
APPLICATION.... DRAWINGS
LEVEL.FUNCTION. 1 I
ID.............
REMARK.........

@ID............ 202309260081340469.10
@ID............ 202309260081340469.10
PROTOCOL.ID.... 202309260081340469.10
PROCESS.DATE... 20230926
TIME.MSECS..... 11:14:29:654
K.USER......... INPUTTER
APPLICATION.... AC.INWARD.ENTRY
LEVEL.FUNCTION. 1
ID.............
REMARK......... ENQUIRY - AC.INTERFACE.REPORT

@ID............ 202309260081340490.06
@ID............ 202309260081340490.06
PROTOCOL.ID.... 202309260081340490.06
PROCESS.DATE... 20230926
TIME.MSECS..... 11:14:50:299
K.USER......... INPUTTER
APPLICATION.... AC.INWARD.ENTRY LIST F.PROTOCOL @ID PROTOCOL.ID PROCESS.DATE TIME.MSECS K.USER APPLICATION LEVEL.FUNCTION ID REMARK PAGE 3 11:34:02 23 NOV 2023

LEVEL.FUNCTION. 1
ID.............
REMARK......... ENQUIRY - AC.INTERFACE.REPORT

@ID............ 202309260081340509.05
@ID............ 202309260081340509.05
PROTOCOL.ID.... 202309260081340509.05
PROCESS.DATE... 20230926
TIME.MSECS..... 11:15:09:201
K.USER......... INPUTTER
APPLICATION.... AC.INWARD.ENTRY
LEVEL.FUNCTION. 1
ID.............
REMARK......... ENQUIRY - AC.INTERFACE.REPORT

@ID............ 202309260081340529.00
@ID............ 202309260081340529.00
PROTOCOL.ID.... 202309260081340529.00
PROCESS.DATE... 20230926
TIME.MSECS..... 11:15:29:015
K.USER......... INPUTTER
APPLICATION.... AC.INWARD.ENTRY
LEVEL.FUNCTION. 1
ID.............
REMARK......... ENQUIRY - AC.INTERFACE.REPORT

@ID............ 202310033834745376.01
@ID............ 202310033834745376.01
PROTOCOL.ID.... 202310033834745376.01
PROCESS.DATE... 20230926
TIME.MSECS..... 12:36:16:380
K.USER......... ASHWIN.KUMAR
APPLICATION.... CATEGORY
LEVEL.FUNCTION. 1 S
ID.............
REMARK.........

@ID............ 202309260081340496.06
@ID............ 202309260081340496.06
PROTOCOL.ID.... 202309260081340496.06
PROCESS.DATE... 20230926
TIME.MSECS..... 11:14:56:370 LIST F.PROTOCOL @ID PROTOCOL.ID PROCESS.DATE TIME.MSECS K.USER APPLICATION LEVEL.FUNCTION ID REMARK PAGE 4 11:34:02 23 NOV 2023

K.USER......... INPUTTER
APPLICATION.... AC.INWARD.ENTRY
LEVEL.FUNCTION. 1
ID.............
REMARK......... ENQUIRY - AC.INTERFACE.REPORT

@ID............ 202310031395145227.00
@ID............ 202310031395145227.00
PROTOCOL.ID.... 202310031395145227.00
PROCESS.DATE... 20230926
TIME.MSECS..... 12:33:47:173
K.USER......... ASHWIN.KUMAR
APPLICATION.... SIGN.ON
LEVEL.FUNCTION.
ID.............
REMARK.........

@ID............ TEST1-70226
@ID............ TEST1-70226
PROTOCOL.ID.... TEST1-70226
PROCESS.DATE... 20230926
TIME.MSECS..... 12:52:55:808
K.USER......... TEST1
APPLICATION.... PGM.BREAK
LEVEL.FUNCTION. 1
ID.............
REMARK.........

@ID............ 202309264115451975.00
@ID............ 202309264115451975.00
PROTOCOL.ID.... 202309264115451975.00
PROCESS.DATE... 20230926
TIME.MSECS..... 14:26:15:315
K.USER......... INPUTTER
APPLICATION.... ENQUIRY.SELECT
LEVEL.FUNCTION. 1
ID............. TRADE.POS.VALUATION_BH0010001_INPUTTER
REMARK......... 1