topic How to fix issue with parsing events (log files)? in Splunk Search

How to fix issue with parsing events (log files)?

SplunkDash — Tue, 09 May 2023 16:24:00 GMT

Hey,

I have issues with parsing events, multiple events/records (raw data) are within the same event. Sample data and my props configuration file are giving below. How help will be highly appreciated. Thank you so much in advance for your help:

Sample Events

May 9, 2023 5:46:00 AM com.vontu.messaging.chainData.PremiseMessageChainTracer beginChain FINER: Message chain #5: Begin processing message [0C369823455-7843-44D7-89E3-SAB21BF361F24F] from [Request].

May 9, 2023 5:46:00 AM com.vontu.messaging.chainData.ComponentProcessor$PerMessageProcessor processMessageComponents FINER: Processing of message [0C369823655-7843-44D7-89E3-B21BF361F24F]:[Unknown] took: 0 ms

May 9, 2023 5:46:00 AM com.vontu.messaging.chain.ComponentProcessor$PerMessageProcessor processMessageComponents FINER: Processing of message [0C369823-7843-44D7-89E3-B21BF361F24F]:[Unknown] took: 0 ms

May 9, 2023 5:46:00 AM com.vontu.messaging.chain.ComponentProcessor$PerMessageProcessor processMessageComponents FINER: Processing of message [0C3698sdss23-7843-44D7-89E3-B21BF361F24F]:[attached-email-body.txt] took: 11 ms

May 9, 2023 5:46:00 AM com.vontu.messaging.chain.ComponentProcessorr$PerMessageProcessor processMessageComponents FINER: Processing of message [0C3698saaa23-7843-44D7-89E3-B21BF361566F24F]:[Unknown] took: 10 ms

May 9, 2023 5:46:00 AM com.vontu.messaging.chain.ComponentProcessor$PerMessageProcessor processMessageComponents FINER: Processing of message [0C3698sdaa23-7843-44D7-89E3-B21BF361F24F]:[[EXT] [LibraryLink] Library Link of the Day for 2023-05-09_attached-email-body] took: 9 ms

May 9, 2023 5:46:00 AM com.vontu.messaging.chain.imagepreclassifier.ImagePreclassifierManager applyPrefiltersOnImages INFO: Skipping component: unknown for image filtering as required component.

PROPS.CONF

[auditrdata]

SHOULD_LINEMERGE=false

LINE_BREAKER=([\r\n]+)\w+\s\d{2},\s\d{4}

NO_BINARY_CHECK=true

CHARSET=UTF-8

disabled=false

TIME_PREFIX=^

TIME_FORMAT=%b %d, %Y %H:%M:%S

MAX_TIMESTAMP_LOOKAHEAD=30

TRUNCATE=5000

Re: How to fix issue with parsing events (log files)?

VatsalJagani — Wed, 10 May 2023 09:05:19 GMT

@SplunkDash - I hope the sourcetype is correct as you said, so try the below configuration:

[auditrdata] SHOULD_LINEMERGE=false LINE_BREAKER=([\r\n]+)\w+\s\d{1,2},\s\d{4} NO_BINARY_CHECK=true TIME_PREFIX=^ TIME_FORMAT=%b %d, %Y %I:%M:%S %p MAX_TIMESTAMP_LOOKAHEAD=30 TRUNCATE=5000

I hope this helps!!! Kindly upvote if it does!!!

Re: How to fix issue with parsing events (log files)?

SplunkDash — Tue, 09 May 2023 18:20:53 GMT

Hello @VatsalJagani ,

Thank you so much for your quick response, truly appreciate it. Now it's parsing one event as 2 events. I think the sample events I provided you should have 2 lines for each event, but the way I copied it looked like one line sorry about that. Each of the event should be as follow. Any recommendations would be highly appreciated. Thank you so much again.

May 9, 2023 5:46:00 AM com.vontu.messaging.chainData.PremiseMessageChainTracer beginChain
FINER: Message chain #5: Begin processing message [0C369823455-7843-44D7-89E3-SAB21BF361F24F] from [Request].

May 9, 2023 5:46:00 AM com.vontu.messaging.chainData.ComponentProcessor PerMessageProcessor processMessageComponents
FINER: Processing of message [0C369823655-7843-44D7-89E3-B21BF361F24F]:[Unknown] took: 0 ms

May 9, 2023 5:46:00 AM com.vontu.messaging.chain.ComponentProcessorPerMessageProcessor processMessageComponents
FINER: Processing of message [0C369823-7843-44D7-89E3-B21BF361F24F]:[Unknown] took: 0 ms

Re: How to fix issue with parsing events (log files)?

m_pham — Tue, 09 May 2023 20:38:22 GMT

Adding to @VatsalJagani 's suggestion, try this:

[auditrdata] SHOULD_LINEMERGE=false LINE_BREAKER=([\r\n]+)\w+\s\d{1,2},\s\d{4} NO_BINARY_CHECK=true TIME_PREFIX=^ TIME_FORMAT=%b %d, %Y %I:%M:%S %p MAX_TIMESTAMP_LOOKAHEAD=30 TRUNCATE=5000

Re: How to fix issue with parsing events (log files)?

goncalocoelho — Tue, 09 May 2023 20:40:37 GMT

Hi,

the problem is in Line Breaker

LINE_BREAKER=([\r\n]+)\w+\s\d{1,2},\s\d{4}

Re: How to fix issue with parsing events (log files)?

SplunkDash — Wed, 10 May 2023 03:49:12 GMT

@VatsalJagani, @goncalocoelho, @m_pham

Thank you so much you all, truly appreciate it. Let me try with this and let you know how it goes. Thank you so much again.

Re: How to fix issue with parsing events (log files)?

VatsalJagani — Wed, 10 May 2023 09:06:17 GMT

@SplunkDash - Just updated my original response based on the change that you asked.

Just updated the LINE_BREAKER to

([\r\n]+)\w+\s\d{1,2},\s\d{4}

Try original response now.

Re: How to fix issue with parsing events (log files)?

SplunkDash — Thu, 11 May 2023 03:11:33 GMT

Hello @VatsalJagani

@m_pham

@goncalocoelho :

Thank you so much again. Now events are parsing without doubling up (Multiple) events within single event. But now issue with the Line that has the TIMESTAMP. Every event is missing Line that has the TIMESTAMP and showing as follow. Any help will be highly appreciated, thank you again.

Showing Now:

FINER: Message chain #5: Begin processing message [0C369823455-7843-44D7-89E3-SAB21BF361F24F] from [Request].

FINER: Processing of message [0C369823655-7843-44D7-89E3-B21BF361F24F]:[Unknown] took: 0 ms

FINER: Processing of message [0C369823-7843-44D7-89E3-B21BF361F24F]:[Unknown] took: 0 ms

Should be:

May 9, 2023 5:46:00 AM com.vontu.messaging.chainData.PremiseMessageChainTracer beginChain
FINER: Message chain #5: Begin processing message [0C369823455-7843-44D7-89E3-SAB21BF361F24F] from [Request].

Re: How to fix issue with parsing events (log files)?

VatsalJagani — Thu, 11 May 2023 05:55:04 GMT

@SplunkDash - Try btool and show config CLI command to see what configuration is placed for this sourcetype and there is no conflicting configuration already present in your Splunk environment.

Re: How to fix issue with parsing events (log files)?

SplunkDash — Thu, 11 May 2023 09:43:36 GMT

Hello @VatsalJagani , @goncalocoelho , @m_pham

Thank you so much again, this is completely new ingestion and no conflict found. Now is one interesting thing here, getting some of the events with the proper structure (with TIMESTAMP Line or no missing line) and some other events without that, thinking there might be issues (or inconsistency) with the format of the TIMESTAMP causing that issue. What you think? If this is the issue, what should I do, any recommendation would be highly appreciated.

Re: How to fix issue with parsing events (log files)?

VatsalJagani — Thu, 11 May 2023 11:25:09 GMT

@SplunkDash - The recommendation is to first find the different format that is causing the issue (or find all the different formats) and then based on that we maybe able to suggest something.

* Also it is unusual for a single system to generate two different timestamp formats for the same data.

* It could be either a different host, or different source.