Regex of .txt file in Splunk

mukhan1 — Tue, 05 Dec 2023 12:32:52 GMT

Hello,

I'm integrating the .txt file in Splunk, however while integrating the file my events are breaking into single line not all events but many of them are breaking into single line. Attaching the log file in comments.

Below is how my data is appearing on Splunk when I add this txt file into Splunk. Is there any way I can limit the starting and ending point of my event. I want my data to be started from @ID and ends on REMARK.

And if I use regex "(@ID[\s\S]*?REMARK[\s\S]*?)(?=@ID|$)" while adding the data, many of my logs are getting missing attaching the snapshot of it also. not sure how to resolve this issue, if anyone can know how i can integrate this .txt file to get my event start from (@ID to REMARK)

Re: Regex of .txt file in Splunk

mukhan1 — Tue, 05 Dec 2023 12:33:49 GMT

[H[2JLIST F.PROTOCOL @ID PROTOCOL.ID PROCESS.DATE TIME.MSECS K.USER APPLICATION LEVEL.FUNCTION ID REMARK PAGE 1 11:34:02 23 NOV 2023

@ID............ 202309260081340532.21
@ID............ 202309260081340532.21
PROTOCOL.ID.... 202309260081340532.21
PROCESS.DATE... 20230926
TIME.MSECS..... 11:15:32:934
K.USER......... INPUTTER
APPLICATION.... AC.INWARD.ENTRY
LEVEL.FUNCTION. 1
ID.............
REMARK......... ENQUIRY - AC.INTERFACE.REPORT

@ID............ 202309260081340523.16
@ID............ 202309260081340523.16
PROTOCOL.ID.... 202309260081340523.16
PROCESS.DATE... 20230926
TIME.MSECS..... 11:15:23:649
K.USER......... INPUTTER
APPLICATION.... AC.INWARD.ENTRY
LEVEL.FUNCTION. 1
ID.............
REMARK......... ENQUIRY - AC.INTERFACE.REPORT

@ID............ 202309260081340465.12
@ID............ 202309260081340465.12
PROTOCOL.ID.... 202309260081340465.12
PROCESS.DATE... 20230926
TIME.MSECS..... 11:14:25:781
K.USER......... INPUTTER
APPLICATION.... AC.INWARD.ENTRY
LEVEL.FUNCTION. 1
ID.............
REMARK......... ENQUIRY - AC.INTERFACE.REPORT

@ID............ AUTHORISER-8232
@ID............ AUTHORISER-8232
PROTOCOL.ID.... AUTHORISER-8232
PROCESS.DATE... 20230926
TIME.MSECS..... 09:08:19:962
K.USER......... AUTHORISER
APPLICATION.... PGM.BREAK
LEVEL.FUNCTION. 1
ID............. [H[2JLIST F.PROTOCOL @ID PROTOCOL.ID PROCESS.DATE TIME.MSECS K.USER APPLICATION LEVEL.FUNCTION ID REMARK PAGE 2 11:34:02 23 NOV 2023

REMARK.........

@ID............ 202309260081340530.06
@ID............ 202309260081340530.06
PROTOCOL.ID.... 202309260081340530.06
PROCESS.DATE... 20230926
TIME.MSECS..... 11:15:30:223
K.USER......... INPUTTER
APPLICATION.... AC.INWARD.ENTRY
LEVEL.FUNCTION. 1
ID.............
REMARK......... ENQUIRY - AC.INTERFACE.REPORT

@ID............ 202309269535047401.01
@ID............ 202309269535047401.01
PROTOCOL.ID.... 202309269535047401.01
PROCESS.DATE... 20230926
TIME.MSECS..... 13:10:01:201
K.USER......... INPUTTER
APPLICATION.... DRAWINGS
LEVEL.FUNCTION. 1 I
ID.............
REMARK.........

@ID............ 202309260081340469.10
@ID............ 202309260081340469.10
PROTOCOL.ID.... 202309260081340469.10
PROCESS.DATE... 20230926
TIME.MSECS..... 11:14:29:654
K.USER......... INPUTTER
APPLICATION.... AC.INWARD.ENTRY
LEVEL.FUNCTION. 1
ID.............
REMARK......... ENQUIRY - AC.INTERFACE.REPORT

@ID............ 202309260081340490.06
@ID............ 202309260081340490.06
PROTOCOL.ID.... 202309260081340490.06
PROCESS.DATE... 20230926
TIME.MSECS..... 11:14:50:299
K.USER......... INPUTTER
APPLICATION.... AC.INWARD.ENTRY [H[2JLIST F.PROTOCOL @ID PROTOCOL.ID PROCESS.DATE TIME.MSECS K.USER APPLICATION LEVEL.FUNCTION ID REMARK PAGE 3 11:34:02 23 NOV 2023

LEVEL.FUNCTION. 1
ID.............
REMARK......... ENQUIRY - AC.INTERFACE.REPORT

@ID............ 202309260081340509.05
@ID............ 202309260081340509.05
PROTOCOL.ID.... 202309260081340509.05
PROCESS.DATE... 20230926
TIME.MSECS..... 11:15:09:201
K.USER......... INPUTTER
APPLICATION.... AC.INWARD.ENTRY
LEVEL.FUNCTION. 1
ID.............
REMARK......... ENQUIRY - AC.INTERFACE.REPORT

@ID............ 202309260081340529.00
@ID............ 202309260081340529.00
PROTOCOL.ID.... 202309260081340529.00
PROCESS.DATE... 20230926
TIME.MSECS..... 11:15:29:015
K.USER......... INPUTTER
APPLICATION.... AC.INWARD.ENTRY
LEVEL.FUNCTION. 1
ID.............
REMARK......... ENQUIRY - AC.INTERFACE.REPORT

@ID............ 202310033834745376.01
@ID............ 202310033834745376.01
PROTOCOL.ID.... 202310033834745376.01
PROCESS.DATE... 20230926
TIME.MSECS..... 12:36:16:380
K.USER......... ASHWIN.KUMAR
APPLICATION.... CATEGORY
LEVEL.FUNCTION. 1 S
ID.............
REMARK.........

@ID............ 202309260081340496.06
@ID............ 202309260081340496.06
PROTOCOL.ID.... 202309260081340496.06
PROCESS.DATE... 20230926
TIME.MSECS..... 11:14:56:370 [H[2JLIST F.PROTOCOL @ID PROTOCOL.ID PROCESS.DATE TIME.MSECS K.USER APPLICATION LEVEL.FUNCTION ID REMARK PAGE 4 11:34:02 23 NOV 2023

K.USER......... INPUTTER
APPLICATION.... AC.INWARD.ENTRY
LEVEL.FUNCTION. 1
ID.............
REMARK......... ENQUIRY - AC.INTERFACE.REPORT

@ID............ 202310031395145227.00
@ID............ 202310031395145227.00
PROTOCOL.ID.... 202310031395145227.00
PROCESS.DATE... 20230926
TIME.MSECS..... 12:33:47:173
K.USER......... ASHWIN.KUMAR
APPLICATION.... SIGN.ON
LEVEL.FUNCTION.
ID.............
REMARK.........

@ID............ TEST1-70226
@ID............ TEST1-70226
PROTOCOL.ID.... TEST1-70226
PROCESS.DATE... 20230926
TIME.MSECS..... 12:52:55:808
K.USER......... TEST1
APPLICATION.... PGM.BREAK
LEVEL.FUNCTION. 1
ID.............
REMARK.........

@ID............ 202309264115451975.00
@ID............ 202309264115451975.00
PROTOCOL.ID.... 202309264115451975.00
PROCESS.DATE... 20230926
TIME.MSECS..... 14:26:15:315
K.USER......... INPUTTER
APPLICATION.... ENQUIRY.SELECT
LEVEL.FUNCTION. 1
ID............. TRADE.POS.VALUATION_BH0010001_INPUTTER
REMARK......... 1

Re: Regex of .txt file in Splunk

PickleRick — Tue, 05 Dec 2023 13:40:20 GMT

Generally, you should avoid using SHOULD_LINEMERGE=true whenever you can.

In your case it seems like something like this (along with SHOULD_LINEMERGE=false) should work

LINE_BREAKER = ^REMARK[^\r\n]+([\r\n]+)@ID

Re: Regex of .txt file in Splunk

mukhan1 — Wed, 06 Dec 2023 06:49:58 GMT

Thanks for the response, when I'm using this line breaker regex alot of events are missing attaching screenshot as reference.
Is there any way i can read that whole file into single payload, If yes this can resolve my issue.

Re: Regex of .txt file in Splunk

PickleRick — Wed, 06 Dec 2023 08:09:30 GMT

That is kinda strange.

If you check it on regex101 - https://regex101.com/r/Bavlui/1 (I have no idea how long the saved regexes are kept) - it seems to work. As you can see, the group 1 is properly matched to the space between events.

So there might be something not 100% copy-pasteable and your events might actually look a bit different (maybe have some hanging spaces/tabs or something like that).

In general, your LINE_BREAKER should match the place on which you want to break the stream into separate events and must contain a capturing group which will match the part which separates one event from another. That group will be discarded as the "spacer" between events.

topic Re: Regex of .txt file in Splunk in Getting Data In

Regex of .txt file in Splunk

Re: Regex of .txt file in Splunk

Re: Regex of .txt file in Splunk

Re: Regex of .txt file in Splunk

Re: Regex of .txt file in Splunk