Hi, I have a flat file multiline log:
Here is my props.conf
[emailAlerts2]
SHOULD_LINEMERGE=true
BREAK_ONLY_BEFORE_DATE = false
BREAK_ONLY_BEFORE = </EcomLogEntry>
TRUNCATE=0
MAX_EVENTS =512
TIME_PREFIX = \d\d\d\d-\d\d-\d\dT\d\d:\d\d:\d\d-0500
MAX_TIMESTAMP_LOOKAHEAD=25
LINE_BREAKER_LOOKBEHIND=500
TIME_PREFIX = ^Date
Sample log file is:
<EcomLogEntry>
Date: 2015-01-13T23:59:00-0500
Date: 1/13/2015 11:59 PM
Log Stuff 1
Log Stuff 2
Log Stuf 3 etc
</EcomLogEntry>
What I want to do is grab everything between the EcomLogEntry tags (tried many different break options). Sometimes it works. Other times, I get just one event with a Date or one event with , it's not consistent.
Any suggestions?
Thank you,
Chris
Try this
[emailAlerts2]
BREAK_ONLY_BEFORE = <EcomLogEntry>
MAX_EVENTS = 512
MAX_TIMESTAMP_LOOKAHEAD = 40
MUST_BREAK_AFTER = </EcomLogEntry>
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = true
TIME_PREFIX = Date\:
TRUNCATE = 0
Hi Chris
Add one more configuration line BREAK_ONLY_BEFORE_DATE = false to your props.conf
[emailAlerts2]
BREAK_ONLY_BEFORE = \<EcomLogEntry\>
NO_BINARY_CHECK = 1
BREAK_ONLY_BEFORE_DATE = false
SHOULD_LINEMERGE = true
TIME_FORMAT = \d\d\d\d-\d\d-\d\dT\d\d:\d\d:\d\d-0500
TIME_PREFIX = Date\:
TRUNCATE = 0
BREAK_ONLY_BEFORE_DATE = false
Try this
[emailAlerts2]
BREAK_ONLY_BEFORE = <EcomLogEntry>
MAX_EVENTS = 512
MAX_TIMESTAMP_LOOKAHEAD = 40
MUST_BREAK_AFTER = </EcomLogEntry>
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = true
TIME_PREFIX = Date\:
TRUNCATE = 0
Thanks for the tips. Still having an issue. My latest config:
[emailAlerts2]
BREAK_ONLY_BEFORE = \<EcomLogEntry\>
NO_BINARY_CHECK = 1
BREAK_ONLY_BEFORE_DATE = false
SHOULD_LINEMERGE = true
TIME_FORMAT = \d\d\d\d-\d\d-\d\dT\d\d:\d\d:\d\d-0500
TIME_PREFIX = Date\:
TRUNCATE = 0
I'm still getting multiple breaks. One on < EcomLogEntry > and the other on the Date. Its like its ignoring the BREAK_ONLY_BEFORE_DATE directive. I also tried added the MUST_BREAK_AFTER = < / EcomLogEntry >, but still had the same results.
Any other tips?
Thank you,
Chris
I guess the TIME_FORMAT needs to be fixed here. Try the below configuration.
BREAK_ONLY_BEFORE = \<EcomLogEntry\>
MAX_EVENTS = 512
MAX_TIMESTAMP_LOOKAHEAD = 150
MUST_BREAK_AFTER = \<\/EcomLogEntry\>
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=true
TIME_FORMAT = %Y-%m-%dT%H:%M:%S%Z
TIME_PREFIX = Date\:
TRUNCATE = 0
Ugh. Still now go. Just so I'm clear. I'm using the Splunk Universal Forwarder and editing the
C:\Program Files\SplunkUniversalForwarder\etc\system\local\props.conf . Then I restart the SUF to pick up the new config changes.
Latest config below. I still see it breaking on the date timestamp. Also, the BREAK_ONLY_BEFORE includes the line < EcomLogEntry >, so at the end of an event, I will get:
< EcomLogEntry >
[emailAlerts2]
BREAK_ONLY_BEFORE = \<EcomLogEntry\>
NO_BINARY_CHECK = 1
BREAK_ONLY_BEFORE_DATE = false
SHOULD_LINEMERGE = true
TIME_FORMAT=%Y-%m-%dT%H:%M:%S%z
TIME_PREFIX = Date\:
TRUNCATE = 0
BREAK_ONLY_BEFORE_DATE = false
Thanks for your Help!
Chris
You should be putting these settings in your indexer(s), not the forwarder.
OMG. That was it. So sorry for the swirl!
The last config looks like its working fine!
Thanks all
Chris
thats cool. Incase if you plan to use Heavy Forwarder between Uni. Forwarder and Indexer, then put these changes in Heavy forwarder aswell.
You have two TIME_PREFIX entries in your props.conf file. Perhaps the first should be TIME_FORMAT?
I think your event start is correct?
Modify the BREAK_ONLY_BEFORE line to
BREAK_ONLY_BEFORE = \<EcomLogEntry\>
Typo? Modify it to what?
Updated now.. anything in between angel brackets without space, will be removed after post in this website.
So,
1. I have put space after & before angel brackets. Please remove them.
2. Put back slash for both angel brackets.. (use escape characters for both angel brackets)
3. As richgalloway pointed, 1st TIME_PREFIX should be TIME_FORMAT
4. Remove the following lines
MAX_TIMESTAMP_LOOKAHEAD=25
LINE_BREAKER_LOOKBEHIND=500
try these steps and let us know.
If you indent a line 4 spaces the editor will ignore angle brackets on that line.