How to edit my current configuration for correct multiline event breaking?

[chrisboy68]

Hi, I have a flat file multiline log:
Here is my props.conf

 [emailAlerts2]
 SHOULD_LINEMERGE=true
 BREAK_ONLY_BEFORE_DATE = false
 BREAK_ONLY_BEFORE = </EcomLogEntry>
 TRUNCATE=0
 MAX_EVENTS =512
 TIME_PREFIX = \d\d\d\d-\d\d-\d\dT\d\d:\d\d:\d\d-0500
 MAX_TIMESTAMP_LOOKAHEAD=25
 LINE_BREAKER_LOOKBEHIND=500
 TIME_PREFIX = ^Date

Sample log file is:

<EcomLogEntry>
Date: 2015-01-13T23:59:00-0500
Date: 1/13/2015 11:59 PM

Log Stuff 1
Log Stuff 2
Log Stuf 3 etc

</EcomLogEntry>

What I want to do is grab everything between the EcomLogEntry tags (tried many different break options). Sometimes it works. Other times, I get just one event with a Date or one event with , it's not consistent.

Any suggestions?

Thank you,

Chris

[jayannah]

Hi Chris

Add one more configuration line BREAK_ONLY_BEFORE_DATE = false to your props.conf

         [emailAlerts2]
         BREAK_ONLY_BEFORE = \<EcomLogEntry\>
         NO_BINARY_CHECK = 1
         BREAK_ONLY_BEFORE_DATE = false
         SHOULD_LINEMERGE = true
         TIME_FORMAT = \d\d\d\d-\d\d-\d\dT\d\d:\d\d:\d\d-0500
         TIME_PREFIX = Date\:
         TRUNCATE = 0
         BREAK_ONLY_BEFORE_DATE = false

[somesoni2 · Accepted solution]

Try this

 [emailAlerts2]
BREAK_ONLY_BEFORE = <EcomLogEntry>
MAX_EVENTS = 512
MAX_TIMESTAMP_LOOKAHEAD = 40
MUST_BREAK_AFTER = </EcomLogEntry>
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = true
TIME_PREFIX = Date\:
TRUNCATE = 0

[chrisboy68]

Thanks for the tips. Still having an issue. My latest config:

    [emailAlerts2]
    BREAK_ONLY_BEFORE = \<EcomLogEntry\>
    NO_BINARY_CHECK = 1
    BREAK_ONLY_BEFORE_DATE = false
    SHOULD_LINEMERGE = true
    TIME_FORMAT = \d\d\d\d-\d\d-\d\dT\d\d:\d\d:\d\d-0500
    TIME_PREFIX = Date\:
    TRUNCATE = 0

I'm still getting multiple breaks. One on < EcomLogEntry > and the other on the Date. Its like its ignoring the BREAK_ONLY_BEFORE_DATE directive. I also tried added the MUST_BREAK_AFTER = < / EcomLogEntry >, but still had the same results.

Any other tips?

Thank you,

Chris

[somesoni2]

I guess the TIME_FORMAT needs to be fixed here. Try the below configuration.

BREAK_ONLY_BEFORE = \<EcomLogEntry\>
MAX_EVENTS = 512
MAX_TIMESTAMP_LOOKAHEAD = 150
MUST_BREAK_AFTER = \<\/EcomLogEntry\>
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=true
TIME_FORMAT = %Y-%m-%dT%H:%M:%S%Z
TIME_PREFIX = Date\:
TRUNCATE = 0

[chrisboy68]

Ugh. Still now go. Just so I'm clear. I'm using the Splunk Universal Forwarder and editing the
C:\Program Files\SplunkUniversalForwarder\etc\system\local\props.conf . Then I restart the SUF to pick up the new config changes.

Latest config below. I still see it breaking on the date timestamp. Also, the BREAK_ONLY_BEFORE includes the line < EcomLogEntry >, so at the end of an event, I will get:

< EcomLogEntry >

    [emailAlerts2]
    BREAK_ONLY_BEFORE = \<EcomLogEntry\>
    NO_BINARY_CHECK = 1
    BREAK_ONLY_BEFORE_DATE = false
    SHOULD_LINEMERGE = true
    TIME_FORMAT=%Y-%m-%dT%H:%M:%S%z
    TIME_PREFIX = Date\:
    TRUNCATE = 0
    BREAK_ONLY_BEFORE_DATE = false

Thanks for your Help!

Chris

[richgalloway]

You should be putting these settings in your indexer(s), not the forwarder.

[chrisboy68]

OMG. That was it. So sorry for the swirl!

The last config looks like its working fine!

Thanks all

Chris

[jayannah]

thats cool. Incase if you plan to use Heavy Forwarder between Uni. Forwarder and Indexer, then put these changes in Heavy forwarder aswell.

[richgalloway]

You have two TIME_PREFIX entries in your props.conf file. Perhaps the first should be TIME_FORMAT?

[jayannah]

I think your event start is correct?

Modify the BREAK_ONLY_BEFORE line to

                       BREAK_ONLY_BEFORE = \<EcomLogEntry\>

[chrisboy68]

Typo? Modify it to what?

[jayannah]

Updated now.. anything in between angel brackets without space, will be removed after post in this website.

So,
1. I have put space after & before angel brackets. Please remove them.
2. Put back slash for both angel brackets.. (use escape characters for both angel brackets)
3. As richgalloway pointed, 1st TIME_PREFIX should be TIME_FORMAT
4. Remove the following lines
MAX_TIMESTAMP_LOOKAHEAD=25
LINE_BREAKER_LOOKBEHIND=500

try these steps and let us know.

[richgalloway]

If you indent a line 4 spaces the editor will ignore angle brackets on that line.