XML Logfile not breaking correctly

[nwales]

I have the below log entry which is getting split at the end_date stanza.

I have MAX_TIMESTAMP_LOOKAHEAD=23 set in my props but it is having no impact. I have tried multiple other things but to no avail. Any help much appreciated!

2014-01-27 16:30:08,411 | TRACE | 1768824013-84717 | ReservationService | beans.LoggingUtilsBean 223 | DWH Body:
<?xml version="1.0" encoding="UTF-8"?>

PM
redacted
redacted
95193
2014-01-27 16:30:08.410
1
EUR
redacted
Success


redacted
p2149148
redacted

[narwhal · Accepted solution]

I got this sorted out with Nick. Turns out he needed two things.

1. props.conf on the indexers, not the LWF/UF's
2. props.conf contents of

BREAK_ ONLY_BEFORE _DATE = true

TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N

TIME_PREFIX = ^

(NOTE THE COMMA, NOT A PERIOD in the TIME_FORMAT --the "unwanted" timestamps have periods before the %3N's)

[lguinn2]

New answer based on comments to my other answer: this solution should work if the entire file should be treated as a single event.

In props.conf on the indexer (or wherever your data is being parsed), try this

[yoursourcetypehere]
TIME_PREFIX = \<end_date>
MAX_TIMESTAMP_LOOKAHEAD = 23
SHOULD_LINEMERGE = false
LINE_BREAKER = ((*FAIL))
TRUNCATE = 99999999

I used the info in this answer Each File as One Single Splunk Event for part of this.

If you want to use the date on the first line as your event date, do this instead:

[yoursourcetypehere]
MAX_TIMESTAMP_LOOKAHEAD = 23
SHOULD_LINEMERGE = false
LINE_BREAKER = ((*FAIL))
TRUNCATE = 99999999

[lguinn2]

In props.conf on the indexer (or wherever your data is being parsed), try this

[yoursourcetypehere]
BREAK_ONLY_BEFORE = \<reservationresults>
MAX_EVENTS = 1024
TIME_PREFIX = \<end_date>
MAX_TIMESTAMP_LOOKAHEAD = 23

You might not need MAX_EVENTS, but the default maximum lines per event is 256, so I usually set it higher. You might also need to add TIME_FORMAT, but I think this is enough information so that Splunk will parse your XML log into proper events.

FYI, you need the \ because the < is a special character in regular expressions.

[chris]

@MuS: thx for clarifying that, reading the documentation helps sometimes

[nwales]

Apologies, I tried to edit my question unsuccessfully all night due to broken captcha's.

To clarify, I only want it to break at the very first line, not in the XML so what you see there should be one event.

[MuS]

nope, MAX_TIMESTAMP_LOOKAHEAD sets a value to tell Splunk how far past the TIME_PREFIX location it must check in the event for the timestamp.

[chris]

Isn't MAX_TIMESTAMP_LOOKAHEAD to small to reach the TIME_PREFIX?