topic Re: How to edit my current configuration for correct multiline event breaking? in Getting Data In

How to edit my current configuration for correct multiline event breaking?

chrisboy68 — Wed, 14 Jan 2015 20:22:41 GMT

Hi, I have a flat file multiline log:
Here is my props.conf

 [emailAlerts2]
 SHOULD_LINEMERGE=true
 BREAK_ONLY_BEFORE_DATE = false
 BREAK_ONLY_BEFORE = </EcomLogEntry>
 TRUNCATE=0
 MAX_EVENTS =512
 TIME_PREFIX = \d\d\d\d-\d\d-\d\dT\d\d:\d\d:\d\d-0500
 MAX_TIMESTAMP_LOOKAHEAD=25
 LINE_BREAKER_LOOKBEHIND=500
 TIME_PREFIX = ^Date

Sample log file is:

<EcomLogEntry>
Date: 2015-01-13T23:59:00-0500
Date: 1/13/2015 11:59 PM

Log Stuff 1
Log Stuff 2
Log Stuf 3 etc

</EcomLogEntry>

What I want to do is grab everything between the EcomLogEntry tags (tried many different break options). Sometimes it works. Other times, I get just one event with a Date or one event with , it's not consistent.

Any suggestions?

Thank you,

Chris

Re: How to edit my current configuration for correct multiline event breaking?

jayannah — Mon, 28 Sep 2020 18:41:06 GMT

I think your event start is correct?

Modify the BREAK_ONLY_BEFORE line to

                       BREAK_ONLY_BEFORE = \<EcomLogEntry\>

Re: How to edit my current configuration for correct multiline event breaking?

chrisboy68 — Wed, 14 Jan 2015 20:39:07 GMT

Typo? Modify it to what?

Re: How to edit my current configuration for correct multiline event breaking?

richgalloway — Mon, 28 Sep 2020 18:38:07 GMT

You have two TIME_PREFIX entries in your props.conf file. Perhaps the first should be TIME_FORMAT?

Re: How to edit my current configuration for correct multiline event breaking?

jayannah — Mon, 28 Sep 2020 18:38:10 GMT

Updated now.. anything in between angel brackets without space, will be removed after post in this website.

So,
1. I have put space after & before angel brackets. Please remove them.
2. Put back slash for both angel brackets.. (use escape characters for both angel brackets)
3. As richgalloway pointed, 1st TIME_PREFIX should be TIME_FORMAT
4. Remove the following lines
MAX_TIMESTAMP_LOOKAHEAD=25
LINE_BREAKER_LOOKBEHIND=500

try these steps and let us know.

Re: How to edit my current configuration for correct multiline event breaking?

somesoni2 — Wed, 14 Jan 2015 23:17:07 GMT

Try this

 [emailAlerts2]
BREAK_ONLY_BEFORE = <EcomLogEntry>
MAX_EVENTS = 512
MAX_TIMESTAMP_LOOKAHEAD = 40
MUST_BREAK_AFTER = </EcomLogEntry>
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = true
TIME_PREFIX = Date\:
TRUNCATE = 0

Re: How to edit my current configuration for correct multiline event breaking?

richgalloway — Wed, 14 Jan 2015 23:19:22 GMT

If you indent a line 4 spaces the editor will ignore angle brackets on that line.

Re: How to edit my current configuration for correct multiline event breaking?

chrisboy68 — Mon, 28 Sep 2020 18:38:24 GMT

Thanks for the tips. Still having an issue. My latest config:

    [emailAlerts2]
    BREAK_ONLY_BEFORE = \<EcomLogEntry\>
    NO_BINARY_CHECK = 1
    BREAK_ONLY_BEFORE_DATE = false
    SHOULD_LINEMERGE = true
    TIME_FORMAT = \d\d\d\d-\d\d-\d\dT\d\d:\d\d:\d\d-0500
    TIME_PREFIX = Date\:
    TRUNCATE = 0

I'm still getting multiple breaks. One on < EcomLogEntry > and the other on the Date. Its like its ignoring the BREAK_ONLY_BEFORE_DATE directive. I also tried added the MUST_BREAK_AFTER = < / EcomLogEntry >, but still had the same results.

Any other tips?

Thank you,

Chris

Re: How to edit my current configuration for correct multiline event breaking?

jayannah — Mon, 28 Sep 2020 18:41:17 GMT

Hi Chris

Add one more configuration line BREAK_ONLY_BEFORE_DATE = false to your props.conf

         [emailAlerts2]
         BREAK_ONLY_BEFORE = \<EcomLogEntry\>
         NO_BINARY_CHECK = 1
         BREAK_ONLY_BEFORE_DATE = false
         SHOULD_LINEMERGE = true
         TIME_FORMAT = \d\d\d\d-\d\d-\d\dT\d\d:\d\d:\d\d-0500
         TIME_PREFIX = Date\:
         TRUNCATE = 0
         BREAK_ONLY_BEFORE_DATE = false

Re: How to edit my current configuration for correct multiline event breaking?

somesoni2 — Thu, 15 Jan 2015 15:58:18 GMT

I guess the TIME_FORMAT needs to be fixed here. Try the below configuration.

BREAK_ONLY_BEFORE = \<EcomLogEntry\>
MAX_EVENTS = 512
MAX_TIMESTAMP_LOOKAHEAD = 150
MUST_BREAK_AFTER = \<\/EcomLogEntry\>
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=true
TIME_FORMAT = %Y-%m-%dT%H:%M:%S%Z
TIME_PREFIX = Date\:
TRUNCATE = 0

Re: How to edit my current configuration for correct multiline event breaking?

chrisboy68 — Mon, 28 Sep 2020 18:38:27 GMT

Ugh. Still now go. Just so I'm clear. I'm using the Splunk Universal Forwarder and editing the
C:\Program Files\SplunkUniversalForwarder\etc\system\local\props.conf . Then I restart the SUF to pick up the new config changes.

Latest config below. I still see it breaking on the date timestamp. Also, the BREAK_ONLY_BEFORE includes the line < EcomLogEntry >, so at the end of an event, I will get:

< EcomLogEntry >

    [emailAlerts2]
    BREAK_ONLY_BEFORE = \<EcomLogEntry\>
    NO_BINARY_CHECK = 1
    BREAK_ONLY_BEFORE_DATE = false
    SHOULD_LINEMERGE = true
    TIME_FORMAT=%Y-%m-%dT%H:%M:%S%z
    TIME_PREFIX = Date\:
    TRUNCATE = 0
    BREAK_ONLY_BEFORE_DATE = false

Thanks for your Help!

Chris

Re: How to edit my current configuration for correct multiline event breaking?

richgalloway — Thu, 15 Jan 2015 17:57:16 GMT

You should be putting these settings in your indexer(s), not the forwarder.

Re: How to edit my current configuration for correct multiline event breaking?

chrisboy68 — Thu, 15 Jan 2015 18:09:18 GMT

OMG. That was it. So sorry for the swirl!

The last config looks like its working fine!

Thanks all

Chris

Re: How to edit my current configuration for correct multiline event breaking?

jayannah — Thu, 15 Jan 2015 21:12:08 GMT

thats cool. Incase if you plan to use Heavy Forwarder between Uni. Forwarder and Indexer, then put these changes in Heavy forwarder aswell.