XML Fields, Multivalues, Extractions?

[mreidy]

Hi All,

I've got a web service/SOAP call generating a file with the following XML output to a file on a regular basis and I want to pull it into Splunk and be able to break it into multiple lines/records. Each time a new file is generated I'd like Splunk to break the file on the <Table> so that each file read ends up generating 8 different lines/records.

I've tried the following settings in props.conf to no avail:

SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE = \<Table\>
BREAK_ONLY_BEFORE_DATE = false
REPORT-xmlext = xml-extr

Any help is MUCH appreciated!

<?xml version="1.0" encoding="utf-8"?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"><soap:Body><GetRecentActivityResponse xmlns="http://tempuri.org/"><GetRecentActivityResult><xs:schema id="tmpDS" xmlns="" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:msdata="urn:schemas-microsoft-com:xml-msdata"><xs:element name="tmpDS" msdata:IsDataSet="true" msdata:UseCurrentLocale="true"><xs:complexType><xs:choice minOccurs="0" maxOccurs="unbounded"><xs:element name="Table"><xs:complexType><xs:sequence><xs:element name="LenderId" type="xs:int" minOccurs="0" /><xs:element name="MMRLenderID" type="xs:string" minOccurs="0" /><xs:element name="Active" type="xs:boolean" minOccurs="0" /><xs:element name="LastAppSent" type="xs:dateTime" minOccurs="0" /><xs:element name="LastAckRecvd" type="xs:dateTime" minOccurs="0" /><xs:element name="LastDecRecvdTS" type="xs:dateTime" minOccurs="0" /><xs:element name="AppCount" type="xs:int" minOccurs="0" /><xs:element name="ACK_Count" type="xs:int" minOccurs="0" /><xs:element name="DEC_Count" type="xs:int" minOccurs="0" /><xs:element name="DecTO_Count" type="xs:int" minOccurs="0" /><xs:element name="ExcessiveDecTO" type="xs:string" minOccurs="0" /><xs:element name="DecWaiting_Count" type="xs:int" minOccurs="0" /><xs:element name="LastDecRecvd" type="xs:string" minOccurs="0" /><xs:element name="NACK_Count" type="xs:int" minOccurs="0" /><xs:element name="ScoreTOCount" type="xs:int" minOccurs="0" /><xs:element name="AckTO_Count" type="xs:int" minOccurs="0" /><xs:element name="ExcessiveAckTO" type="xs:string" minOccurs="0" /></xs:sequence></xs:complexType></xs:element></xs:choice></xs:complexType></xs:element></xs:schema><diffgr:diffgram xmlns:msdata="urn:schemas-microsoft-com:xml-msdata" xmlns:diffgr="urn:schemas-microsoft-com:xml-diffgram-v1"><tmpDS xmlns=""><Table diffgr:id="Table1" msdata:rowOrder="0"><LenderId>1</LenderId><MMRLenderID>FNC</MMRLenderID><Active>true</Active><LastAppSent>2012-03-02T10:07:44.46-05:00</LastAppSent><LastAckRecvd>2012-03-02T10:07:48.09-05:00</LastAckRecvd><LastDecRecvdTS>2012-03-02T10:08:35.933-05:00</LastDecRecvdTS><AppCount>95</AppCount><ACK_Count>93</ACK_Count><DEC_Count>91</DEC_Count><DecTO_Count>0</DecTO_Count><ExcessiveDecTO>No</ExcessiveDecTO><DecWaiting_Count>2</DecWaiting_Count><LastDecRecvd>APPROVE</LastDecRecvd><NACK_Count>0</NACK_Count><ScoreTOCount>1</ScoreTOCount><AckTO_Count>0</AckTO_Count><ExcessiveAckTO>No</ExcessiveAckTO></Table><Table diffgr:id="Table2" msdata:rowOrder="1"><LenderId>3</LenderId><MMRLenderID>CAP</MMRLenderID><Active>true</Active><LastAppSent>2012-03-02T10:07:21.42-05:00</LastAppSent><LastAckRecvd>2012-03-02T10:07:36.183-05:00</LastAckRecvd><LastDecRecvdTS>2012-03-02T10:07:48.343-05:00</LastDecRecvdTS><AppCount>46</AppCount><ACK_Count>46</ACK_Count><DEC_Count>49</DEC_Count><DecTO_Count>0</DecTO_Count><ExcessiveDecTO>No</ExcessiveDecTO><DecWaiting_Count>0</DecWaiting_Count><LastDecRecvd>APPROVE</LastDecRecvd><NACK_Count>0</NACK_Count><ScoreTOCount>0</ScoreTOCount><AckTO_Count>0</AckTO_Count><ExcessiveAckTO>No</ExcessiveAckTO></Table><Table diffgr:id="Table3" msdata:rowOrder="2"><LenderId>4</LenderId><MMRLenderID>SAN</MMRLenderID><Active>true</Active><LastAppSent>2012-03-02T10:07:21.43-05:00</LastAppSent><LastAckRecvd>2012-03-02T10:07:27.38-05:00</LastAckRecvd><LastDecRecvdTS>2012-03-02T10:07:31.793-05:00</LastDecRecvdTS><AppCount>60</AppCount><ACK_Count>61</ACK_Count><DEC_Count>67</DEC_Count><DecTO_Count>0</DecTO_Count><ExcessiveDecTO>No</ExcessiveDecTO><DecWaiting_Count>0</DecWaiting_Count><LastDecRecvd>DECLINE</LastDecRecvd><NACK_Count>0</NACK_Count><ScoreTOCount>0</ScoreTOCount><AckTO_Count>0</AckTO_Count><ExcessiveAckTO>No</ExcessiveAckTO></Table><Table diffgr:id="Table4" msdata:rowOrder="3"><LenderId>6</LenderId><MMRLenderID>WFS</MMRLenderID><Active>true</Active><LastAppSent>2012-03-02T10:07:05.687-05:00</LastAppSent><LastAckRecvd>2012-03-02T10:07:09.293-05:00</LastAckRecvd><LastDecRecvdTS>2012-03-02T10:08:45.723-05:00</LastDecRecvdTS><AppCount>45</AppCount><ACK_Count>41</ACK_Count><DEC_Count>40</DEC_Count><DecTO_Count>0</DecTO_Count><ExcessiveDecTO>No</ExcessiveDecTO><DecWaiting_Count>3</DecWaiting_Count><LastDecRecvd>DECLINE</LastDecRecvd><NACK_Count>0</NACK_Count><ScoreTOCount>0</ScoreTOCount><AckTO_Count>0</AckTO_Count><ExcessiveAckTO>No</ExcessiveAckTO></Table><Table diffgr:id="Table5" msdata:rowOrder="4"><LenderId>7</LenderId><MMRLenderID>DRV</MMRLenderID><Active>true</Active><LastAppSent>2012-03-02T10:08:14.983-05:00</LastAppSent><LastAckRecvd>2012-03-02T10:08:24.27-05:00</LastAckRecvd><LastDecRecvdTS>2012-03-02T10:08:22.197-05:00</LastDecRecvdTS><AppCount>55</AppCount><ACK_Count>55</ACK_Count><DEC_Count>59</DEC_Count><DecTO_Count>0</DecTO_Count><ExcessiveDecTO>No</ExcessiveDecTO><DecWaiting_Count>0</DecWaiting_Count><LastDecRecvd>DECLINE</LastDecRecvd><NACK_Count>0</NACK_Count><ScoreTOCount>0</ScoreTOCount><AckTO_Count>0</AckTO_Count><ExcessiveAckTO>No</ExcessiveAckTO></Table><Table diffgr:id="Table6" msdata:rowOrder="5"><LenderId>11</LenderId><MMRLenderID>CHO</MMRLenderID><Active>true</Active><LastAppSent>2012-03-02T00:24:55.433-05:00</LastAppSent><LastAckRecvd>2012-03-02T10:02:23.147-05:00</LastAckRecvd><LastDecRecvdTS>2012-03-02T10:02:38.12-05:00</LastDecRecvdTS><AppCount>6</AppCount><ACK_Count>7</ACK_Count><DEC_Count>18</DEC_Count><DecWaiting_Count>0</DecWaiting_Count><LastDecRecvd>APPROVE</LastDecRecvd><NACK_Count>0</NACK_Count><ScoreTOCount>0</ScoreTOCount></Table><Table diffgr:id="Table7" msdata:rowOrder="6"><LenderId>12</LenderId><MMRLenderID>ACA</MMRLenderID><Active>true</Active><LastAppSent>2012-03-02T10:06:53.473-05:00</LastAppSent><LastAckRecvd>2012-03-02T10:08:37.967-05:00</LastAckRecvd><LastDecRecvdTS>2012-03-02T09:55:07.05-05:00</LastDecRecvdTS><AppCount>10</AppCount><ACK_Count>10</ACK_Count><DEC_Count>7</DEC_Count><DecTO_Count>0</DecTO_Count><ExcessiveDecTO>No</ExcessiveDecTO><DecWaiting_Count>3</DecWaiting_Count><LastDecRecvd>DECLINE</LastDecRecvd><NACK_Count>0</NACK_Count><ScoreTOCount>0</ScoreTOCount><AckTO_Count>0</AckTO_Count><ExcessiveAckTO>No</ExcessiveAckTO></Table><Table diffgr:id="Table8" msdata:rowOrder="7"><LenderId>13</LenderId><MMRLenderID>WST</MMRLenderID><Active>true</Active><LastAppSent>2012-03-02T09:54:20.237-05:00</LastAppSent><LastAckRecvd>2012-03-02T09:54:35.747-05:00</LastAckRecvd><LastDecRecvdTS>2012-03-02T09:55:01.103-05:00</LastDecRecvdTS><AppCount>2</AppCount><ACK_Count>2</ACK_Count><DEC_Count>2</DEC_Count><DecTO_Count>0</DecTO_Count><ExcessiveDecTO>No</ExcessiveDecTO><DecWaiting_Count>0</DecWaiting_Count><LastDecRecvd>APPCOND</LastDecRecvd><NACK_Count>0</NACK_Count><ScoreTOCount>0</ScoreTOCount><AckTO_Count>0</AckTO_Count><ExcessiveAckTO>No</ExcessiveAckTO></Table></tmpDS></diffgr:diffgram></GetRecentActivityResult></GetRecentActivityResponse></soap:Body></soap:Envelope>

[Ayn]

As I understand it this is all a single line?

There are two concepts that come into play here:

1. What Splunk considers to be a "line".
2. What Splunk considers to be an "event".

1 is defined upon according to the LINE_BREAKER directive in props.conf (default is ([\r\n]+)).
2 is defined by the various line merging settings.

So, first Splunk decides what a line is, then it decides how to merge lines into events. Therefore, to have an event for each <Table> section you need to define a LINE_BREAKER that tells Splunk to break on that. The tricky thing is, LINE_BREAKER requires a matching group in its regex, and Splunk will remove the text that is matched! This answer http://splunk-base.splunk.com/answers/358/is-it-possible-to-tell-line_breaker-to-stop-eating-my-angl... has some details on how to deal with that.

[Ayn]

Sorry, typo - I meant LINE_BREAKER = (</Table>) of course.

[Ayn]

LINE_BREAKER = LINEBREAKER = [\>\s]((?=\<table\>))

Typo? (Re the LINEBREAKER after the first equals sign)

Also you don't need to escape the tags. I suggest starting with something that should be guaranteed to break the line, like simply LINE_BREAKER = </Table>. Then work your way from there.

[mreidy]

Yes, the xml data is all on a single line.

I've tried the following in my props.conf:

SHOULD_LINEMERGE = false

LINE_BREAKER = LINEBREAKER = >\s

But it's still not splitting into more than one event. I tried restarting Splunk too.