topic Re: How to keep multiple events in one XML file but pull the key-value pairs from each transaction line? in Dashboards & Visualizations

How to keep multiple events in one XML file but pull the key-value pairs from each transaction line?

thesba — Sun, 23 Nov 2014 00:30:57 GMT

I have thousands of XML files which are being indexed in Splunk and I would like to extract values from the XML. The XML data contains product information and transactions every time a sale occurs. However, when I used xmlkv to extract the values from the XML, if there are multiple transactions, Splunk only sees the last key-value pair and disregards that multiple items were in the bucket/shopping cart. I've played with xmlsplit and reviewed the documents regarding splitting the XML file in the props.conf, however then each event would lose valuable information like the timestamp and location which isn't stored in each transactionline. An example of the data is below:

<JournalHeader>
</JournalHeader>
<SaleEvent>
  <TransactionDetailGroup>
    <TransactionLine status="normal">
      <ItemLine>
        <ItemCode>
          <POSCode>1001</POSCode>
        </ItemCode>
        <Description>Product A</Description>
      </ItemLine>
    </TransactionLine>
    <TransactionLine status="normal">
      <ItemLine>
        <ItemCode>
          <POSCode>1011</POSCode>
        </ItemCode>
        <Description>Product B</Description>
      </ItemLine>
    </TransactionLine>
    <TransactionLine status="normal">
      <ItemLine>
        <ItemCode>
          <POSCode>1021</POSCode>
        </ItemCode>
        <Description>Product C</Description>
      </ItemLine>
    </TransactionLine>
    <TransactionLine status="normal">
    ...
    </TransactionLine>
  </TransactionDetailGroup>
  <TransactionSummary>
...
  </TransactionSummary>
</SaleEvent>

If I do a table of 100 events and list the POSCode and Description, I would get 100 events with either 1001 & Product A or 1021 & Product C.

Is there a way to keep the event as a single event but drill down into each TransactionLine and pull the key-values from each?

Re: How to keep multiple events in one XML file but pull the key-value pairs from each transaction line?

martin_mueller — Sun, 23 Nov 2014 11:52:45 GMT

Try using spath: http://docs.splunk.com/Documentation/Splunk/6.2.0/SearchReference/spath

your base search | spath

Re: How to keep multiple events in one XML file but pull the key-value pairs from each transaction line?

thesba — Mon, 24 Nov 2014 04:56:15 GMT

spath is helpful, thank you!. What I'm trying to do is calculate sums on the purchased quantity and product. With spath the fields have values of

Qty
1
2
1

Description
Product A
Product B
Product C

How would I associate
Product A, Qty 1
Product B, Qty 2
Product C, Qty 1

So I could do a search on all events and determine total products sold during different time periods? I've tried using mvjoin and eval?

Re: How to keep multiple events in one XML file but pull the key-value pairs from each transaction line?

martin_mueller — Tue, 25 Nov 2014 00:27:17 GMT

Doing that with XML is a bit annoying... with JSON you can apply spath partially, mvexpand, and apply the second half of spath then. Here you can do something like this:

... | spath | eval temp = mvzip(Description, Qty) | mvexpand temp | rex field=temp "(?<Description>[^,]+),(?<Qty>[^,]+)" | fields - temp

That's assuming the description doesn't contain a comma.

Re: How to keep multiple events in one XML file but pull the key-value pairs from each transaction line?

jensonthottian — Sun, 26 Apr 2015 22:47:25 GMT

So @thesba finally what is the solution you undertake?
Can you share the approach, I am also trying to parse through POS XML log files.