topic Re: XML Logfile not breaking correctly in Dashboards & Visualizations

XML Logfile not breaking correctly

nwales — Mon, 28 Sep 2020 15:44:01 GMT

I have the below log entry which is getting split at the end_date stanza.

I have MAX_TIMESTAMP_LOOKAHEAD=23 set in my props but it is having no impact. I have tried multiple other things but to no avail. Any help much appreciated!

Re: XML Logfile not breaking correctly

lguinn2 — Tue, 28 Jan 2014 06:59:52 GMT

In props.conf on the indexer (or wherever your data is being parsed), try this

[yoursourcetypehere]
BREAK_ONLY_BEFORE = \<reservationresults>
MAX_EVENTS = 1024
TIME_PREFIX = \<end_date>
MAX_TIMESTAMP_LOOKAHEAD = 23

You might not need MAX_EVENTS, but the default maximum lines per event is 256, so I usually set it higher. You might also need to add TIME_FORMAT, but I think this is enough information so that Splunk will parse your XML log into proper events.

FYI, you need the \ because the < is a special character in regular expressions.

Re: XML Logfile not breaking correctly

chris — Mon, 28 Sep 2020 15:44:13 GMT

Isn't MAX_TIMESTAMP_LOOKAHEAD to small to reach the TIME_PREFIX?

Re: XML Logfile not breaking correctly

MuS — Mon, 28 Sep 2020 15:44:16 GMT

nope, MAX_TIMESTAMP_LOOKAHEAD sets a value to tell Splunk how far past the TIME_PREFIX location it must check in the event for the timestamp.

Re: XML Logfile not breaking correctly

nwales — Tue, 28 Jan 2014 15:12:28 GMT

Apologies, I tried to edit my question unsuccessfully all night due to broken captcha's.

To clarify, I only want it to break at the very first line, not in the XML so what you see there should be one event.

Re: XML Logfile not breaking correctly

lguinn2 — Tue, 28 Jan 2014 22:05:45 GMT

New answer based on comments to my other answer: this solution should work if the entire file should be treated as a single event.

In props.conf on the indexer (or wherever your data is being parsed), try this

[yoursourcetypehere]
TIME_PREFIX = \<end_date>
MAX_TIMESTAMP_LOOKAHEAD = 23
SHOULD_LINEMERGE = false
LINE_BREAKER = ((*FAIL))
TRUNCATE = 99999999

I used the info in this answer Each File as One Single Splunk Event for part of this.

If you want to use the date on the first line as your event date, do this instead:

[yoursourcetypehere]
MAX_TIMESTAMP_LOOKAHEAD = 23
SHOULD_LINEMERGE = false
LINE_BREAKER = ((*FAIL))
TRUNCATE = 99999999

Re: XML Logfile not breaking correctly

chris — Wed, 29 Jan 2014 07:02:55 GMT

@MuS: thx for clarifying that, reading the documentation helps sometimes

Re: XML Logfile not breaking correctly

narwhal — Thu, 13 Feb 2014 22:45:10 GMT

I got this sorted out with Nick. Turns out he needed two things.

props.conf on the indexers, not the LWF/UF's
props.conf contents of

BREAK_ ONLY_BEFORE _DATE = true

TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N

TIME_PREFIX = ^

(NOTE THE COMMA, NOT A PERIOD in the TIME_FORMAT --the "unwanted" timestamps have periods before the %3N's)