TIME_FORMAT and XML

[colinj]

Howdy,

I'm running in to a problem getting some XML to parse in to events properly. The log has multi-line entries as you might expect and in the XML are date strings that I want Splunk to ignore. So, each log entry begins with the following time stamp (which I want to use):

[03/12/12 16:15:30.103]: <Some log data goes here>
<possibly followed by a bunch of XML>
[03/12/12 16:15:30.112]: <Some more log data goes here>

The entries in the log file might only a single line long or might be as long as several hundred lines of XML before the next so-called real time spamp. Part of the XML data being generated are time strings like this: 20120312161445.247Z, which I don't want.

What I want is for all of the lines between one time stamp and the next to be grouped in to a single event.

Here's what I have in my props.conf file:

TIME_PREFIX = \[
TIME_FORMAT = %m/%d/%y %H:%M:%S.%3N

However Splunk is still splitting on the other time stamps within the XML. So, what am I doing wrong?

[kristian_kolb · Accepted solution]

Are you editing the correct props.conf? (i.e. where the parsing occurs).
If you have a Universal Forwarder -> Indexer OR Lightweight Forwarder -> Indexer setup, the props.conf to edit is on the Indexer.

If you are using a full/heavy forwarder, the settings should go there.

Apart from that your settings look good. You could possibly further qualify your TIME_PREFIX regex with (if tour 'real' timestamps are the actually in the beginning of the line);

TIME_PREFIX=^\[

How have you configured linebreaking? I'd recommend that you use

SHOULD_LINEMERGE=false
LINE_BREAKER = ([\r\n]+)\[\d\d

which explicitly tells splunk to break events after a newline followed by the beginning of a 'real' timestamp.

Hope this helps. If not, please give us more of your props.conf, and perhaps some more sample events.

/Kristian

[colinj]

Thanks! That did the trick.