Getting Data In

Regex of .txt file in Splunk

mukhan1
Explorer

Hello,

I'm integrating the .txt file in Splunk, however while integrating the file my events are breaking into single line not all events but many of them are breaking into single line. Attaching the log file in comments.

Below is how my data is appearing on Splunk when I add this txt file into Splunk. Is there any way I can limit the starting and ending point of my event. I want my data to be started from @ID and ends on REMARK. 

mukhan1_0-1701778570243.png 
And if I use regex "(@ID[\s\S]*?REMARK[\s\S]*?)(?=@ID|$)" while adding the data, many of my logs are getting missing attaching the snapshot of it also. not sure how to resolve this issue,  if anyone can know how i can integrate this .txt file to get my event start from (@ID to REMARK)

 

 

mukhan1_1-1701779229791.png

Labels (3)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

Generally, you should avoid using SHOULD_LINEMERGE=true whenever you can.

In your case it seems  like something like this (along with SHOULD_LINEMERGE=false) should work

LINE_BREAKER = ^REMARK[^\r\n]+([\r\n]+)@ID

0 Karma

mukhan1
Explorer

Thanks for the response, when I'm using this line breaker regex  alot of events are missing attaching screenshot as reference.
Is there any way i can read that whole file into single payload, If yes this can resolve my issue. 

mukhan1_0-1701845375950.png

 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

That is kinda strange.

If you check it on regex101 - https://regex101.com/r/Bavlui/1 (I have no idea how long the saved regexes are kept) - it seems to work. As you can see, the group 1 is properly matched to the space between events.

So there might be something not 100% copy-pasteable and your events might actually look a bit different (maybe have some hanging spaces/tabs or something like that).

In general, your LINE_BREAKER should match the place on which you want to break the stream into separate events and must contain a capturing group which will match the part which separates one event from another. That group will be discarded as the "spacer" between events.

0 Karma

mukhan1
Explorer

LIST F.PROTOCOL @ID PROTOCOL.ID PROCESS.DATE TIME.MSECS K.USER APPLICATION LEVEL.FUNCTION ID REMARK PAGE 1 11:34:02 23 NOV 2023

@ID............ 202309260081340532.21
@ID............ 202309260081340532.21
PROTOCOL.ID.... 202309260081340532.21
PROCESS.DATE... 20230926
TIME.MSECS..... 11:15:32:934
K.USER......... INPUTTER
APPLICATION.... AC.INWARD.ENTRY
LEVEL.FUNCTION. 1
ID.............
REMARK......... ENQUIRY - AC.INTERFACE.REPORT

@ID............ 202309260081340523.16
@ID............ 202309260081340523.16
PROTOCOL.ID.... 202309260081340523.16
PROCESS.DATE... 20230926
TIME.MSECS..... 11:15:23:649
K.USER......... INPUTTER
APPLICATION.... AC.INWARD.ENTRY
LEVEL.FUNCTION. 1
ID.............
REMARK......... ENQUIRY - AC.INTERFACE.REPORT

@ID............ 202309260081340465.12
@ID............ 202309260081340465.12
PROTOCOL.ID.... 202309260081340465.12
PROCESS.DATE... 20230926
TIME.MSECS..... 11:14:25:781
K.USER......... INPUTTER
APPLICATION.... AC.INWARD.ENTRY
LEVEL.FUNCTION. 1
ID.............
REMARK......... ENQUIRY - AC.INTERFACE.REPORT

@ID............ AUTHORISER-8232
@ID............ AUTHORISER-8232
PROTOCOL.ID.... AUTHORISER-8232
PROCESS.DATE... 20230926
TIME.MSECS..... 09:08:19:962
K.USER......... AUTHORISER
APPLICATION.... PGM.BREAK
LEVEL.FUNCTION. 1
ID............. LIST F.PROTOCOL @ID PROTOCOL.ID PROCESS.DATE TIME.MSECS K.USER APPLICATION LEVEL.FUNCTION ID REMARK PAGE 2 11:34:02 23 NOV 2023

REMARK.........

@ID............ 202309260081340530.06
@ID............ 202309260081340530.06
PROTOCOL.ID.... 202309260081340530.06
PROCESS.DATE... 20230926
TIME.MSECS..... 11:15:30:223
K.USER......... INPUTTER
APPLICATION.... AC.INWARD.ENTRY
LEVEL.FUNCTION. 1
ID.............
REMARK......... ENQUIRY - AC.INTERFACE.REPORT

@ID............ 202309269535047401.01
@ID............ 202309269535047401.01
PROTOCOL.ID.... 202309269535047401.01
PROCESS.DATE... 20230926
TIME.MSECS..... 13:10:01:201
K.USER......... INPUTTER
APPLICATION.... DRAWINGS
LEVEL.FUNCTION. 1 I
ID.............
REMARK.........

@ID............ 202309260081340469.10
@ID............ 202309260081340469.10
PROTOCOL.ID.... 202309260081340469.10
PROCESS.DATE... 20230926
TIME.MSECS..... 11:14:29:654
K.USER......... INPUTTER
APPLICATION.... AC.INWARD.ENTRY
LEVEL.FUNCTION. 1
ID.............
REMARK......... ENQUIRY - AC.INTERFACE.REPORT

@ID............ 202309260081340490.06
@ID............ 202309260081340490.06
PROTOCOL.ID.... 202309260081340490.06
PROCESS.DATE... 20230926
TIME.MSECS..... 11:14:50:299
K.USER......... INPUTTER
APPLICATION.... AC.INWARD.ENTRY LIST F.PROTOCOL @ID PROTOCOL.ID PROCESS.DATE TIME.MSECS K.USER APPLICATION LEVEL.FUNCTION ID REMARK PAGE 3 11:34:02 23 NOV 2023

LEVEL.FUNCTION. 1
ID.............
REMARK......... ENQUIRY - AC.INTERFACE.REPORT

@ID............ 202309260081340509.05
@ID............ 202309260081340509.05
PROTOCOL.ID.... 202309260081340509.05
PROCESS.DATE... 20230926
TIME.MSECS..... 11:15:09:201
K.USER......... INPUTTER
APPLICATION.... AC.INWARD.ENTRY
LEVEL.FUNCTION. 1
ID.............
REMARK......... ENQUIRY - AC.INTERFACE.REPORT

@ID............ 202309260081340529.00
@ID............ 202309260081340529.00
PROTOCOL.ID.... 202309260081340529.00
PROCESS.DATE... 20230926
TIME.MSECS..... 11:15:29:015
K.USER......... INPUTTER
APPLICATION.... AC.INWARD.ENTRY
LEVEL.FUNCTION. 1
ID.............
REMARK......... ENQUIRY - AC.INTERFACE.REPORT

@ID............ 202310033834745376.01
@ID............ 202310033834745376.01
PROTOCOL.ID.... 202310033834745376.01
PROCESS.DATE... 20230926
TIME.MSECS..... 12:36:16:380
K.USER......... ASHWIN.KUMAR
APPLICATION.... CATEGORY
LEVEL.FUNCTION. 1 S
ID.............
REMARK.........

@ID............ 202309260081340496.06
@ID............ 202309260081340496.06
PROTOCOL.ID.... 202309260081340496.06
PROCESS.DATE... 20230926
TIME.MSECS..... 11:14:56:370 LIST F.PROTOCOL @ID PROTOCOL.ID PROCESS.DATE TIME.MSECS K.USER APPLICATION LEVEL.FUNCTION ID REMARK PAGE 4 11:34:02 23 NOV 2023

K.USER......... INPUTTER
APPLICATION.... AC.INWARD.ENTRY
LEVEL.FUNCTION. 1
ID.............
REMARK......... ENQUIRY - AC.INTERFACE.REPORT

@ID............ 202310031395145227.00
@ID............ 202310031395145227.00
PROTOCOL.ID.... 202310031395145227.00
PROCESS.DATE... 20230926
TIME.MSECS..... 12:33:47:173
K.USER......... ASHWIN.KUMAR
APPLICATION.... SIGN.ON
LEVEL.FUNCTION.
ID.............
REMARK.........

@ID............ TEST1-70226
@ID............ TEST1-70226
PROTOCOL.ID.... TEST1-70226
PROCESS.DATE... 20230926
TIME.MSECS..... 12:52:55:808
K.USER......... TEST1
APPLICATION.... PGM.BREAK
LEVEL.FUNCTION. 1
ID.............
REMARK.........

@ID............ 202309264115451975.00
@ID............ 202309264115451975.00
PROTOCOL.ID.... 202309264115451975.00
PROCESS.DATE... 20230926
TIME.MSECS..... 14:26:15:315
K.USER......... INPUTTER
APPLICATION.... ENQUIRY.SELECT
LEVEL.FUNCTION. 1
ID............. TRADE.POS.VALUATION_BH0010001_INPUTTER
REMARK......... 1

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...