How to fix issue with parsing events (log files)?

[SplunkDash]

Hey,

I have issues with parsing events, multiple events/records (raw data) are within the same event. Sample data and my props configuration file are giving below. How help will be highly appreciated. Thank you so much in advance for your help:

Sample Events

May 9, 2023 5:46:00 AM com.vontu.messaging.chainData.PremiseMessageChainTracer beginChain FINER: Message chain #5: Begin processing message [0C369823455-7843-44D7-89E3-SAB21BF361F24F] from [Request].

May 9, 2023 5:46:00 AM com.vontu.messaging.chainData.ComponentProcessor$PerMessageProcessor processMessageComponents FINER: Processing of message [0C369823655-7843-44D7-89E3-B21BF361F24F]:[Unknown] took: 0 ms

May 9, 2023 5:46:00 AM com.vontu.messaging.chain.ComponentProcessor$PerMessageProcessor processMessageComponents FINER: Processing of message [0C369823-7843-44D7-89E3-B21BF361F24F]:[Unknown] took: 0 ms

May 9, 2023 5:46:00 AM com.vontu.messaging.chain.ComponentProcessor$PerMessageProcessor processMessageComponents FINER: Processing of message [0C3698sdss23-7843-44D7-89E3-B21BF361F24F]:[attached-email-body.txt] took: 11 ms

May 9, 2023 5:46:00 AM com.vontu.messaging.chain.ComponentProcessorr$PerMessageProcessor processMessageComponents FINER: Processing of message [0C3698saaa23-7843-44D7-89E3-B21BF361566F24F]:[Unknown] took: 10 ms

May 9, 2023 5:46:00 AM com.vontu.messaging.chain.ComponentProcessor$PerMessageProcessor processMessageComponents FINER: Processing of message [0C3698sdaa23-7843-44D7-89E3-B21BF361F24F]:[[EXT] [LibraryLink] Library Link of the Day for 2023-05-09_attached-email-body] took: 9 ms

May 9, 2023 5:46:00 AM com.vontu.messaging.chain.imagepreclassifier.ImagePreclassifierManager applyPrefiltersOnImages INFO: Skipping component: unknown for image filtering as required component.

PROPS.CONF

[auditrdata]

SHOULD_LINEMERGE=false

LINE_BREAKER=([\r\n]+)\w+\s\d{2},\s\d{4}

NO_BINARY_CHECK=true

CHARSET=UTF-8

disabled=false

TIME_PREFIX=^

TIME_FORMAT=%b %d, %Y %H:%M:%S

MAX_TIMESTAMP_LOOKAHEAD=30

TRUNCATE=5000

[goncalocoelho]

Hi, 

the problem is in Line Breaker

LINE_BREAKER=([\r\n]+)\w+\s\d{1,2},\s\d{4}

[VatsalJagani]

@SplunkDash - I hope the sourcetype is correct as you said, so try the below configuration:

[auditrdata]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)\w+\s\d{1,2},\s\d{4}
NO_BINARY_CHECK=true
TIME_PREFIX=^
TIME_FORMAT=%b %d, %Y %I:%M:%S %p
MAX_TIMESTAMP_LOOKAHEAD=30
TRUNCATE=5000

I hope this helps!!! Kindly upvote if it does!!!

[SplunkDash]

Hello @VatsalJagani ,

Thank you so much for your quick response, truly appreciate it. Now it's parsing one event as 2 events. I think the sample events I provided you should have 2 lines for each event, but the way I copied it looked like one line sorry about that. Each of the event should be as follow. Any recommendations would be highly appreciated. Thank you so much again.

May 9, 2023 5:46:00 AM com.vontu.messaging.chainData.PremiseMessageChainTracer beginChain 
FINER: Message chain #5: Begin processing message [0C369823455-7843-44D7-89E3-SAB21BF361F24F] from [Request].

May 9, 2023 5:46:00 AM com.vontu.messaging.chainData.ComponentProcessor PerMessageProcessor processMessageComponents 
FINER: Processing of message [0C369823655-7843-44D7-89E3-B21BF361F24F]:[Unknown] took: 0 ms

May 9, 2023 5:46:00 AM com.vontu.messaging.chain.ComponentProcessorPerMessageProcessor processMessageComponents
FINER: Processing of message [0C369823-7843-44D7-89E3-B21BF361F24F]:[Unknown] took: 0 ms

[VatsalJagani]

@SplunkDash - Just updated my original response based on the change that you asked.

Just updated the LINE_BREAKER to 

([\r\n]+)\w+\s\d{1,2},\s\d{4}

Try original response now.

[m_pham · Accepted solution]

Adding to @VatsalJagani 's suggestion, try this:

[auditrdata]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)\w+\s\d{1,2},\s\d{4}
NO_BINARY_CHECK=true
TIME_PREFIX=^
TIME_FORMAT=%b %d, %Y %I:%M:%S %p
MAX_TIMESTAMP_LOOKAHEAD=30
TRUNCATE=5000

[SplunkDash]

Hello @VatsalJagani

@m_pham 

@goncalocoelho : 

Thank you so much again. Now events are parsing without doubling up (Multiple) events within single event. But now issue with the Line that has the TIMESTAMP. Every event is missing Line that has the TIMESTAMP and showing as follow. Any help will be highly appreciated, thank you again.

Showing Now:

FINER: Message chain #5: Begin processing message [0C369823455-7843-44D7-89E3-SAB21BF361F24F] from [Request].

FINER: Processing of message [0C369823655-7843-44D7-89E3-B21BF361F24F]:[Unknown] took: 0 ms

FINER: Processing of message [0C369823-7843-44D7-89E3-B21BF361F24F]:[Unknown] took: 0 ms

Should be:

May 9, 2023 5:46:00 AM com.vontu.messaging.chainData.PremiseMessageChainTracer beginChain 
FINER: Message chain #5: Begin processing message [0C369823455-7843-44D7-89E3-SAB21BF361F24F] from [Request].

May 9, 2023 5:46:00 AM com.vontu.messaging.chainData.ComponentProcessor PerMessageProcessor processMessageComponents 
FINER: Processing of message [0C369823655-7843-44D7-89E3-B21BF361F24F]:[Unknown] took: 0 ms

May 9, 2023 5:46:00 AM com.vontu.messaging.chain.ComponentProcessorPerMessageProcessor processMessageComponents
FINER: Processing of message [0C369823-7843-44D7-89E3-B21BF361F24F]:[Unknown] took: 0 ms

[VatsalJagani]

@SplunkDash - Try btool and show config CLI command to see what configuration is placed for this sourcetype and there is no conflicting configuration already present in your Splunk environment.

[SplunkDash]

Hello @VatsalJagani , @goncalocoelho , @m_pham 

Thank you so much again, this is completely new ingestion and no conflict found. Now is one interesting thing here, getting some of the events with the proper structure (with TIMESTAMP Line or no missing line) and some other events without that, thinking there might be issues (or inconsistency) with the format of the TIMESTAMP causing that issue. What you think? If this is the issue, what should I do, any recommendation would be highly  appreciated.

[VatsalJagani]

@SplunkDash - The recommendation is to first find the different format that is causing the issue (or find all the different formats) and then based on that we maybe able to suggest something.

* Also it is unusual for a single system to generate two different timestamp formats for the same data.

* It could be either a different host, or different source.

[SplunkDash]

@VatsalJagani, @goncalocoelho, @m_pham 

Thank you so much you all, truly appreciate it. Let me try with this and let you know how it goes. Thank you so much again.