Getting Data In

Parsing of events using regex

mukhan1
Explorer

Hello community,

Below is my sample log file I want to extract each individual piece of event(starting from @ID to REMARK) from the log file. I tried to achieve this by using following regex: (^@ID[\s\S]*?REMARK.*$)
This regex is taking the whole log file as single event. Attaching the snapshot below. 
Also tried to alter the props.conf by using the same regex:

props.conf
[t24]
SHOULD_LINEMERGE=False
LINE_BREAKER=(^@ID[\s\S]*?REMARK.*$)
NO_BINARY_CHECK=true
disabled=false
INDEXED_EXTRACTIONS = csv

 

LIST F.PROTOCOL @ID PROTOCOL.ID PROCESS.DATE TIME.MSECS K.USER APPLICATION LEVEL.FUNCTION ID REMARK                                                                         PAGE    1 11:34:02  23 NOV 2023

@ID............ 202309260081340532.21                                                                                                                                                                           
@ID............ 202309260081340532.21                                                                                                                                                                           
PROTOCOL.ID.... 202309260081340532.21                                                                                                                                                                           
PROCESS.DATE... 20230926                                                                                                                                                                                        
TIME.MSECS..... 11:15:32:934                                                                                                                                                                                    
K.USER......... INPUTTER                                                                                                                                                                                        
APPLICATION.... AC.INWARD.ENTRY                                                                                                                                                                                 
LEVEL.FUNCTION. 1                                                                                                                                                                                               
ID.............                                                                                                                                                                                                 
REMARK......... ENQUIRY - AC.INTERFACE.REPORT                                                                                                                                                                   

@ID............ 202309260081340523.16                                                                                                                                                                           
@ID............ 202309260081340523.16                                                                                                                                                                           
PROTOCOL.ID.... 202309260081340523.16                                                                                                                                                                           
PROCESS.DATE... 20230926                                                                                                                                                                                        
TIME.MSECS..... 11:15:23:649                                                                                                                                                                                    
K.USER......... INPUTTER                                                                                                                                                                                        
APPLICATION.... AC.INWARD.ENTRY                                                                                                                                                                                 
LEVEL.FUNCTION. 1                                                                                                                                                                                               
ID.............                                                                                                                                                                                                 
REMARK......... ENQUIRY - AC.INTERFACE.REPORT 

 

Attaching the screenshot of the data which I'm getting on Splunk by using the regex mentioned above.

mukhan1_3-1701175303903.png

Also attaching the snapshot of regex result which i have checked earlier online.

mukhan1_2-1701175257291.png

I want my data to be shown in table form following is the example snapshot of how I want my data to be appear on Splunk.

mukhan1_4-1701175806186.jpeg

 

Labels (3)
0 Karma

mukhan1
Explorer

Hello @ITWhisperer,
Thanks for the response the provided solution is working however if I edit this file and add more data will this search work same as it is working now? This file keeps getting updated after some time, 

0 Karma

mukhan1
Explorer

Hello @ITWhisperer,
 
Thanks for your response. I was trying to add the search you have provided but failed to get desired value, can you please elaborate further how to use this solution you provided. 

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Try it one line at a time and see where it fails

0 Karma

mukhan1
Explorer

It is not failing since we haven’t define any sourcetype or source in search, we hardcoded the payload, however it is working properly giving the exact same headings required. The problem is how it will extract this information from given sourcetype or source.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust
| makeresults
| eval _raw="LIST F.PROTOCOL @ID PROTOCOL.ID PROCESS.DATE TIME.MSECS K.USER APPLICATION LEVEL.FUNCTION ID REMARK                                                                         PAGE    1 11:34:02  23 NOV 2023

@ID............ 202309260081340532.2
@ID............ 202309260081340532.21
PROTOCOL.ID.... 202309260081340532.21
PROCESS.DATE... 20230926
TIME.MSECS..... 11:15:32:934
K.USER......... INPUTTER
APPLICATION.... AC.INWARD.ENTRY
LEVEL.FUNCTION. 1
ID............. 
REMARK......... ENQUIRY - AC.INTERFACE.REPORT

@ID............ 202309260081340523.16
@ID............ 202309260081340523.16
PROTOCOL.ID.... 202309260081340523.16
PROCESS.DATE... 20230926
TIME.MSECS..... 11:15:23:649
K.USER......... INPUTTER
APPLICATION.... AC.INWARD.ENTRY
LEVEL.FUNCTION. 1
ID............. 
REMARK......... ENQUIRY - AC.INTERFACE.REPORT"
``` The lines above set up sample data in line with your example ```
| rex max_match=0 "(?ms)(?<event>^\@ID.*?REMARK.*?$)"
| mvexpand event
| rex max_match=0 field=event "(?m)(?<namevalue>.+\.+\s.*$)"
| streamstats count as row
| mvexpand namevalue
| rex field=namevalue "(?<name>[^\s]+(?<!\.))\.*?\s(?<value>.*$)"
| eval {name}=value
| fields - name value namevalue event
| stats values(*) as * by row
| fields - row
Get Updates on the Splunk Community!

New Dates, New City: Save the Date for .conf25!

Wake up, babe! New .conf25 dates AND location just dropped!! That's right, this year, .conf25 is taking place ...

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...