topic Re: How can we index an entire XML document as one event? in Dashboards & Visualizations

How can we index an entire XML document as one event?

ddrillic — Wed, 02 Aug 2017 16:16:06 GMT

We have data as XML documents. How can we index each XML document as one Splunk event?

A sample -

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<mlcpMetricsModel xmlns="http://xxxxxxx">
    <canonical>xxxxxx</canonical>
    <duration>PT41M3.401S</duration>
    <env>PROD</env>
    <ecmProcDateTime>20170727_M</ecmProcDateTime>
    <outputRecords>4948262</outputRecords>
    <outputRecordsCommitted>4948262</outputRecordsCommitted>
    <outputRecordsFailed>0</outputRecordsFailed>
    <reportDate>2017-08-02T10:49:31</reportDate>
    <source>CDB</source>
    <startTime>2017-08-02T10:08:18.512</startTime>

</mlcpMetricsModel>

Re: How can we index an entire XML document as one event?

somesoni2 — Wed, 02 Aug 2017 16:21:35 GMT

Try this

[yoursourcetype]
LINE_BREAKER = ([\r\n]+)(?=\<mlcpMetricsModel )
SHOULD_LINEMERGE = false
TIME_PREFIX = \<reportDate\>
TIME_FORMAT = %Y-%m-%dT%H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 19

Re: How can we index an entire XML document as one event?

niketn — Wed, 02 Aug 2017 16:56:04 GMT

@somesoni2, I think \<startTime\> is a better candidate for TimeStamp. However, @ddrillic must confirm.

Re: How can we index an entire XML document as one event?

ddrillic — Wed, 02 Aug 2017 19:14:38 GMT

Thank you both - let me check...

Re: How can we index an entire XML document as one event?

ddrillic — Mon, 07 Aug 2017 15:19:20 GMT

Gorgeous - thank you !! we ended up extracting the XML fields using a series such as - spath mlcpMetricsModel.env ...

Re: How can we index an entire XML document as one event?

ddrillic — Wed, 11 Apr 2018 17:29:36 GMT

Now that I'm preparing for the admin certification, I wonder why @somesoni2 set MAX_TIMESTAMP_LOOKAHEAD = 19, which obviously works but based on the data it appears that the value should be much higher.

Re: How can we index an entire XML document as one event?

somesoni2 — Wed, 11 Apr 2018 17:51:57 GMT

The timestamp is extracted from <reportDate> tag which is 2017-08-02T10:49:31 , 19 character long value.

Re: How can we index an entire XML document as one event?

ddrillic — Wed, 11 Apr 2018 20:40:38 GMT

got it ; -) so, if TIME_PREFIX exists it starts from there, otherwise, from the beginning of the line.

Re: How can we index an entire XML document as one event?

somesoni2 — Wed, 11 Apr 2018 20:50:08 GMT

(thumbs up)

Just in case it's still confusing for anyone, it's the length of timestamp string represented by TIME_FORMAT string. (%Y-%m-%dT%H:%M:%S => %Y(4)-(1)%m(2)-(1)%d(2)T(1)%H(2):(1)%M(2):(1)%S(2) => 4+1+2+1+2+1+2+1+2+1+2 =19)