topic Re: Parsing of events using regex in Getting Data In

Parsing of events using regex

mukhan1 — Tue, 28 Nov 2023 13:06:46 GMT

Hello community,

Below is my sample log file I want to extract each individual piece of event(starting from @ID to REMARK) from the log file. I tried to achieve this by using following regex: (^@ID[\s\S]*?REMARK.*$)
This regex is taking the whole log file as single event. Attaching the snapshot below.
Also tried to alter the props.conf by using the same regex:

props.conf [t24] SHOULD_LINEMERGE=False LINE_BREAKER=(^@ID[\s\S]*?REMARK.*$) NO_BINARY_CHECK=true disabled=false INDEXED_EXTRACTIONS = csv

[H[2JLIST F.PROTOCOL @ID PROTOCOL.ID PROCESS.DATE TIME.MSECS K.USER APPLICATION LEVEL.FUNCTION ID REMARK PAGE 1 11:34:02 23 NOV 2023 @ID............ 202309260081340532.21 @ID............ 202309260081340532.21 PROTOCOL.ID.... 202309260081340532.21 PROCESS.DATE... 20230926 TIME.MSECS..... 11:15:32:934 K.USER......... INPUTTER APPLICATION.... AC.INWARD.ENTRY LEVEL.FUNCTION. 1 ID............. REMARK......... ENQUIRY - AC.INTERFACE.REPORT @ID............ 202309260081340523.16 @ID............ 202309260081340523.16 PROTOCOL.ID.... 202309260081340523.16 PROCESS.DATE... 20230926 TIME.MSECS..... 11:15:23:649 K.USER......... INPUTTER APPLICATION.... AC.INWARD.ENTRY LEVEL.FUNCTION. 1 ID............. REMARK......... ENQUIRY - AC.INTERFACE.REPORT

Attaching the screenshot of the data which I'm getting on Splunk by using the regex mentioned above.

Also attaching the snapshot of regex result which i have checked earlier online.

I want my data to be shown in table form following is the example snapshot of how I want my data to be appear on Splunk.

Re: Parsing of events using regex

ITWhisperer — Tue, 28 Nov 2023 13:31:13 GMT

| makeresults | eval _raw="[H[2JLIST F.PROTOCOL @ID PROTOCOL.ID PROCESS.DATE TIME.MSECS K.USER APPLICATION LEVEL.FUNCTION ID REMARK PAGE 1 11:34:02 23 NOV 2023 @ID............ 202309260081340532.2 @ID............ 202309260081340532.21 PROTOCOL.ID.... 202309260081340532.21 PROCESS.DATE... 20230926 TIME.MSECS..... 11:15:32:934 K.USER......... INPUTTER APPLICATION.... AC.INWARD.ENTRY LEVEL.FUNCTION. 1 ID............. REMARK......... ENQUIRY - AC.INTERFACE.REPORT @ID............ 202309260081340523.16 @ID............ 202309260081340523.16 PROTOCOL.ID.... 202309260081340523.16 PROCESS.DATE... 20230926 TIME.MSECS..... 11:15:23:649 K.USER......... INPUTTER APPLICATION.... AC.INWARD.ENTRY LEVEL.FUNCTION. 1 ID............. REMARK......... ENQUIRY - AC.INTERFACE.REPORT" ``` The lines above set up sample data in line with your example ``` | rex max_match=0 "(?ms)(?<event>^\@ID.*?REMARK.*?$)" | mvexpand event | rex max_match=0 field=event "(?m)(?<namevalue>.+\.+\s.*$)" | streamstats count as row | mvexpand namevalue | rex field=namevalue "(?<name>[^\s]+(?<!\.))\.*?\s(?<value>.*$)" | eval {name}=value | fields - name value namevalue event | stats values(*) as * by row | fields - row

Re: Parsing of events using regex

mukhan1 — Tue, 28 Nov 2023 13:44:46 GMT

Hello @ITWhisperer,

Thanks for your response. I was trying to add the search you have provided but failed to get desired value, can you please elaborate further how to use this solution you provided.

Re: Parsing of events using regex

ITWhisperer — Tue, 28 Nov 2023 13:57:36 GMT

Try it one line at a time and see where it fails

Re: Parsing of events using regex

mukhan1 — Tue, 28 Nov 2023 14:03:40 GMT

Hello @ITWhisperer,
Thanks for the response the provided solution is working however if I edit this file and add more data will this search work same as it is working now? This file keeps getting updated after some time,

Re: Parsing of events using regex

mukhan1 — Tue, 28 Nov 2023 15:59:21 GMT

It is not failing since we haven’t define any sourcetype or source in search, we hardcoded the payload, however it is working properly giving the exact same headings required. The problem is how it will extract this information from given sourcetype or source.