Dashboards & Visualizations

XML Fields, Multivalues, Extractions?

mreidy
New Member

Hi All,

I've got a web service/SOAP call generating a file with the following XML output to a file on a regular basis and I want to pull it into Splunk and be able to break it into multiple lines/records. Each time a new file is generated I'd like Splunk to break the file on the <Table> so that each file read ends up generating 8 different lines/records.

I've tried the following settings in props.conf to no avail:

SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE = \<Table\>
BREAK_ONLY_BEFORE_DATE = false
REPORT-xmlext = xml-extr

Any help is MUCH appreciated!

<?xml version="1.0" encoding="utf-8"?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"><soap:Body><GetRecentActivityResponse xmlns="http://tempuri.org/"><GetRecentActivityResult><xs:schema id="tmpDS" xmlns="" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:msdata="urn:schemas-microsoft-com:xml-msdata"><xs:element name="tmpDS" msdata:IsDataSet="true" msdata:UseCurrentLocale="true"><xs:complexType><xs:choice minOccurs="0" maxOccurs="unbounded"><xs:element name="Table"><xs:complexType><xs:sequence><xs:element name="LenderId" type="xs:int" minOccurs="0" /><xs:element name="MMRLenderID" type="xs:string" minOccurs="0" /><xs:element name="Active" type="xs:boolean" minOccurs="0" /><xs:element name="LastAppSent" type="xs:dateTime" minOccurs="0" /><xs:element name="LastAckRecvd" type="xs:dateTime" minOccurs="0" /><xs:element name="LastDecRecvdTS" type="xs:dateTime" minOccurs="0" /><xs:element name="AppCount" type="xs:int" minOccurs="0" /><xs:element name="ACK_Count" type="xs:int" minOccurs="0" /><xs:element name="DEC_Count" type="xs:int" minOccurs="0" /><xs:element name="DecTO_Count" type="xs:int" minOccurs="0" /><xs:element name="ExcessiveDecTO" type="xs:string" minOccurs="0" /><xs:element name="DecWaiting_Count" type="xs:int" minOccurs="0" /><xs:element name="LastDecRecvd" type="xs:string" minOccurs="0" /><xs:element name="NACK_Count" type="xs:int" minOccurs="0" /><xs:element name="ScoreTOCount" type="xs:int" minOccurs="0" /><xs:element name="AckTO_Count" type="xs:int" minOccurs="0" /><xs:element name="ExcessiveAckTO" type="xs:string" minOccurs="0" /></xs:sequence></xs:complexType></xs:element></xs:choice></xs:complexType></xs:element></xs:schema><diffgr:diffgram xmlns:msdata="urn:schemas-microsoft-com:xml-msdata" xmlns:diffgr="urn:schemas-microsoft-com:xml-diffgram-v1"><tmpDS xmlns=""><Table diffgr:id="Table1" msdata:rowOrder="0"><LenderId>1</LenderId><MMRLenderID>FNC</MMRLenderID><Active>true</Active><LastAppSent>2012-03-02T10:07:44.46-05:00</LastAppSent><LastAckRecvd>2012-03-02T10:07:48.09-05:00</LastAckRecvd><LastDecRecvdTS>2012-03-02T10:08:35.933-05:00</LastDecRecvdTS><AppCount>95</AppCount><ACK_Count>93</ACK_Count><DEC_Count>91</DEC_Count><DecTO_Count>0</DecTO_Count><ExcessiveDecTO>No</ExcessiveDecTO><DecWaiting_Count>2</DecWaiting_Count><LastDecRecvd>APPROVE</LastDecRecvd><NACK_Count>0</NACK_Count><ScoreTOCount>1</ScoreTOCount><AckTO_Count>0</AckTO_Count><ExcessiveAckTO>No</ExcessiveAckTO></Table><Table diffgr:id="Table2" msdata:rowOrder="1"><LenderId>3</LenderId><MMRLenderID>CAP</MMRLenderID><Active>true</Active><LastAppSent>2012-03-02T10:07:21.42-05:00</LastAppSent><LastAckRecvd>2012-03-02T10:07:36.183-05:00</LastAckRecvd><LastDecRecvdTS>2012-03-02T10:07:48.343-05:00</LastDecRecvdTS><AppCount>46</AppCount><ACK_Count>46</ACK_Count><DEC_Count>49</DEC_Count><DecTO_Count>0</DecTO_Count><ExcessiveDecTO>No</ExcessiveDecTO><DecWaiting_Count>0</DecWaiting_Count><LastDecRecvd>APPROVE</LastDecRecvd><NACK_Count>0</NACK_Count><ScoreTOCount>0</ScoreTOCount><AckTO_Count>0</AckTO_Count><ExcessiveAckTO>No</ExcessiveAckTO></Table><Table diffgr:id="Table3" msdata:rowOrder="2"><LenderId>4</LenderId><MMRLenderID>SAN</MMRLenderID><Active>true</Active><LastAppSent>2012-03-02T10:07:21.43-05:00</LastAppSent><LastAckRecvd>2012-03-02T10:07:27.38-05:00</LastAckRecvd><LastDecRecvdTS>2012-03-02T10:07:31.793-05:00</LastDecRecvdTS><AppCount>60</AppCount><ACK_Count>61</ACK_Count><DEC_Count>67</DEC_Count><DecTO_Count>0</DecTO_Count><ExcessiveDecTO>No</ExcessiveDecTO><DecWaiting_Count>0</DecWaiting_Count><LastDecRecvd>DECLINE</LastDecRecvd><NACK_Count>0</NACK_Count><ScoreTOCount>0</ScoreTOCount><AckTO_Count>0</AckTO_Count><ExcessiveAckTO>No</ExcessiveAckTO></Table><Table diffgr:id="Table4" msdata:rowOrder="3"><LenderId>6</LenderId><MMRLenderID>WFS</MMRLenderID><Active>true</Active><LastAppSent>2012-03-02T10:07:05.687-05:00</LastAppSent><LastAckRecvd>2012-03-02T10:07:09.293-05:00</LastAckRecvd><LastDecRecvdTS>2012-03-02T10:08:45.723-05:00</LastDecRecvdTS><AppCount>45</AppCount><ACK_Count>41</ACK_Count><DEC_Count>40</DEC_Count><DecTO_Count>0</DecTO_Count><ExcessiveDecTO>No</ExcessiveDecTO><DecWaiting_Count>3</DecWaiting_Count><LastDecRecvd>DECLINE</LastDecRecvd><NACK_Count>0</NACK_Count><ScoreTOCount>0</ScoreTOCount><AckTO_Count>0</AckTO_Count><ExcessiveAckTO>No</ExcessiveAckTO></Table><Table diffgr:id="Table5" msdata:rowOrder="4"><LenderId>7</LenderId><MMRLenderID>DRV</MMRLenderID><Active>true</Active><LastAppSent>2012-03-02T10:08:14.983-05:00</LastAppSent><LastAckRecvd>2012-03-02T10:08:24.27-05:00</LastAckRecvd><LastDecRecvdTS>2012-03-02T10:08:22.197-05:00</LastDecRecvdTS><AppCount>55</AppCount><ACK_Count>55</ACK_Count><DEC_Count>59</DEC_Count><DecTO_Count>0</DecTO_Count><ExcessiveDecTO>No</ExcessiveDecTO><DecWaiting_Count>0</DecWaiting_Count><LastDecRecvd>DECLINE</LastDecRecvd><NACK_Count>0</NACK_Count><ScoreTOCount>0</ScoreTOCount><AckTO_Count>0</AckTO_Count><ExcessiveAckTO>No</ExcessiveAckTO></Table><Table diffgr:id="Table6" msdata:rowOrder="5"><LenderId>11</LenderId><MMRLenderID>CHO</MMRLenderID><Active>true</Active><LastAppSent>2012-03-02T00:24:55.433-05:00</LastAppSent><LastAckRecvd>2012-03-02T10:02:23.147-05:00</LastAckRecvd><LastDecRecvdTS>2012-03-02T10:02:38.12-05:00</LastDecRecvdTS><AppCount>6</AppCount><ACK_Count>7</ACK_Count><DEC_Count>18</DEC_Count><DecWaiting_Count>0</DecWaiting_Count><LastDecRecvd>APPROVE</LastDecRecvd><NACK_Count>0</NACK_Count><ScoreTOCount>0</ScoreTOCount></Table><Table diffgr:id="Table7" msdata:rowOrder="6"><LenderId>12</LenderId><MMRLenderID>ACA</MMRLenderID><Active>true</Active><LastAppSent>2012-03-02T10:06:53.473-05:00</LastAppSent><LastAckRecvd>2012-03-02T10:08:37.967-05:00</LastAckRecvd><LastDecRecvdTS>2012-03-02T09:55:07.05-05:00</LastDecRecvdTS><AppCount>10</AppCount><ACK_Count>10</ACK_Count><DEC_Count>7</DEC_Count><DecTO_Count>0</DecTO_Count><ExcessiveDecTO>No</ExcessiveDecTO><DecWaiting_Count>3</DecWaiting_Count><LastDecRecvd>DECLINE</LastDecRecvd><NACK_Count>0</NACK_Count><ScoreTOCount>0</ScoreTOCount><AckTO_Count>0</AckTO_Count><ExcessiveAckTO>No</ExcessiveAckTO></Table><Table diffgr:id="Table8" msdata:rowOrder="7"><LenderId>13</LenderId><MMRLenderID>WST</MMRLenderID><Active>true</Active><LastAppSent>2012-03-02T09:54:20.237-05:00</LastAppSent><LastAckRecvd>2012-03-02T09:54:35.747-05:00</LastAckRecvd><LastDecRecvdTS>2012-03-02T09:55:01.103-05:00</LastDecRecvdTS><AppCount>2</AppCount><ACK_Count>2</ACK_Count><DEC_Count>2</DEC_Count><DecTO_Count>0</DecTO_Count><ExcessiveDecTO>No</ExcessiveDecTO><DecWaiting_Count>0</DecWaiting_Count><LastDecRecvd>APPCOND</LastDecRecvd><NACK_Count>0</NACK_Count><ScoreTOCount>0</ScoreTOCount><AckTO_Count>0</AckTO_Count><ExcessiveAckTO>No</ExcessiveAckTO></Table></tmpDS></diffgr:diffgram></GetRecentActivityResult></GetRecentActivityResponse></soap:Body></soap:Envelope>

Tags (2)
0 Karma

Ayn
Legend

As I understand it this is all a single line?

There are two concepts that come into play here:

  1. What Splunk considers to be a "line".
  2. What Splunk considers to be an "event".

1 is defined upon according to the LINE_BREAKER directive in props.conf (default is ([\r\n]+)).
2 is defined by the various line merging settings.

So, first Splunk decides what a line is, then it decides how to merge lines into events. Therefore, to have an event for each <Table> section you need to define a LINE_BREAKER that tells Splunk to break on that. The tricky thing is, LINE_BREAKER requires a matching group in its regex, and Splunk will remove the text that is matched! This answer http://splunk-base.splunk.com/answers/358/is-it-possible-to-tell-line_breaker-to-stop-eating-my-angl... has some details on how to deal with that.

0 Karma

Ayn
Legend

Sorry, typo - I meant LINE_BREAKER = (</Table>) of course.

0 Karma

Ayn
Legend

LINE_BREAKER = LINEBREAKER = [\>\s]((?=\<table\>))

Typo? (Re the LINEBREAKER after the first equals sign)

Also you don't need to escape the tags. I suggest starting with something that should be guaranteed to break the line, like simply LINE_BREAKER = </Table>. Then work your way from there.

0 Karma

mreidy
New Member

Yes, the xml data is all on a single line.

I've tried the following in my props.conf:


SHOULD_LINEMERGE = false
LINE_BREAKER = LINEBREAKER = >\s

But it's still not splitting into more than one event. I tried restarting Splunk too.

0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...