Getting Data In

How to edit my current configuration for correct multiline event breaking?

chrisboy68
Contributor

Hi, I have a flat file multiline log:
Here is my props.conf

 [emailAlerts2]
 SHOULD_LINEMERGE=true
 BREAK_ONLY_BEFORE_DATE = false
 BREAK_ONLY_BEFORE = </EcomLogEntry>
 TRUNCATE=0
 MAX_EVENTS =512
 TIME_PREFIX = \d\d\d\d-\d\d-\d\dT\d\d:\d\d:\d\d-0500
 MAX_TIMESTAMP_LOOKAHEAD=25
 LINE_BREAKER_LOOKBEHIND=500
 TIME_PREFIX = ^Date

Sample log file is:

<EcomLogEntry>
Date: 2015-01-13T23:59:00-0500
Date: 1/13/2015 11:59 PM

Log Stuff 1
Log Stuff 2
Log Stuf 3 etc

</EcomLogEntry>

What I want to do is grab everything between the EcomLogEntry tags (tried many different break options). Sometimes it works. Other times, I get just one event with a Date or one event with , it's not consistent.

Any suggestions?

Thank you,

Chris

0 Karma
1 Solution

somesoni2
Revered Legend

Try this

 [emailAlerts2]
BREAK_ONLY_BEFORE = <EcomLogEntry>
MAX_EVENTS = 512
MAX_TIMESTAMP_LOOKAHEAD = 40
MUST_BREAK_AFTER = </EcomLogEntry>
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = true
TIME_PREFIX = Date\:
TRUNCATE = 0

View solution in original post

jayannah
Builder

Hi Chris

Add one more configuration line BREAK_ONLY_BEFORE_DATE = false to your props.conf

         [emailAlerts2]
         BREAK_ONLY_BEFORE = \<EcomLogEntry\>
         NO_BINARY_CHECK = 1
         BREAK_ONLY_BEFORE_DATE = false
         SHOULD_LINEMERGE = true
         TIME_FORMAT = \d\d\d\d-\d\d-\d\dT\d\d:\d\d:\d\d-0500
         TIME_PREFIX = Date\:
         TRUNCATE = 0
         BREAK_ONLY_BEFORE_DATE = false
0 Karma

somesoni2
Revered Legend

Try this

 [emailAlerts2]
BREAK_ONLY_BEFORE = <EcomLogEntry>
MAX_EVENTS = 512
MAX_TIMESTAMP_LOOKAHEAD = 40
MUST_BREAK_AFTER = </EcomLogEntry>
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = true
TIME_PREFIX = Date\:
TRUNCATE = 0

View solution in original post

chrisboy68
Contributor

Thanks for the tips. Still having an issue. My latest config:

    [emailAlerts2]
    BREAK_ONLY_BEFORE = \<EcomLogEntry\>
    NO_BINARY_CHECK = 1
    BREAK_ONLY_BEFORE_DATE = false
    SHOULD_LINEMERGE = true
    TIME_FORMAT = \d\d\d\d-\d\d-\d\dT\d\d:\d\d:\d\d-0500
    TIME_PREFIX = Date\:
    TRUNCATE = 0

I'm still getting multiple breaks. One on < EcomLogEntry > and the other on the Date. Its like its ignoring the BREAK_ONLY_BEFORE_DATE directive. I also tried added the MUST_BREAK_AFTER = < / EcomLogEntry >, but still had the same results.

Any other tips?

Thank you,

Chris

0 Karma

somesoni2
Revered Legend

I guess the TIME_FORMAT needs to be fixed here. Try the below configuration.

BREAK_ONLY_BEFORE = \<EcomLogEntry\>
MAX_EVENTS = 512
MAX_TIMESTAMP_LOOKAHEAD = 150
MUST_BREAK_AFTER = \<\/EcomLogEntry\>
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=true
TIME_FORMAT = %Y-%m-%dT%H:%M:%S%Z
TIME_PREFIX = Date\:
TRUNCATE = 0

chrisboy68
Contributor

Ugh. Still now go. Just so I'm clear. I'm using the Splunk Universal Forwarder and editing the
C:\Program Files\SplunkUniversalForwarder\etc\system\local\props.conf . Then I restart the SUF to pick up the new config changes.

Latest config below. I still see it breaking on the date timestamp. Also, the BREAK_ONLY_BEFORE includes the line < EcomLogEntry >, so at the end of an event, I will get:

< EcomLogEntry >

    [emailAlerts2]
    BREAK_ONLY_BEFORE = \<EcomLogEntry\>
    NO_BINARY_CHECK = 1
    BREAK_ONLY_BEFORE_DATE = false
    SHOULD_LINEMERGE = true
    TIME_FORMAT=%Y-%m-%dT%H:%M:%S%z
    TIME_PREFIX = Date\:
    TRUNCATE = 0
    BREAK_ONLY_BEFORE_DATE = false

Thanks for your Help!

Chris

0 Karma

richgalloway
SplunkTrust
SplunkTrust

You should be putting these settings in your indexer(s), not the forwarder.

---
If this reply helps you, an upvote would be appreciated.

chrisboy68
Contributor

OMG. That was it. So sorry for the swirl!

The last config looks like its working fine!

Thanks all

Chris

0 Karma

jayannah
Builder

thats cool. Incase if you plan to use Heavy Forwarder between Uni. Forwarder and Indexer, then put these changes in Heavy forwarder aswell.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

You have two TIME_PREFIX entries in your props.conf file. Perhaps the first should be TIME_FORMAT?

---
If this reply helps you, an upvote would be appreciated.
0 Karma

jayannah
Builder

I think your event start is correct?

Modify the BREAK_ONLY_BEFORE line to

                       BREAK_ONLY_BEFORE = \<EcomLogEntry\>
0 Karma

chrisboy68
Contributor

Typo? Modify it to what?

0 Karma

jayannah
Builder

Updated now.. anything in between angel brackets without space, will be removed after post in this website.

So,
1. I have put space after & before angel brackets. Please remove them.
2. Put back slash for both angel brackets.. (use escape characters for both angel brackets)
3. As richgalloway pointed, 1st TIME_PREFIX should be TIME_FORMAT
4. Remove the following lines
MAX_TIMESTAMP_LOOKAHEAD=25
LINE_BREAKER_LOOKBEHIND=500

try these steps and let us know.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

If you indent a line 4 spaces the editor will ignore angle brackets on that line.

---
If this reply helps you, an upvote would be appreciated.
0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!