Is it possible to ingest XML?

nick405060 — Wed, 20 Nov 2019 00:28:53 GMT

It is 2019 and there is still not a comprehensive Splunk Answer or Documentation on how to ingest XML.

Can someone explain to me how to configure props to ingest

<?xml version="1.0" encoding="utf-8"?>
  <ArrayOfUser xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <User>
      <Id>removed</Id>
      <Uuid>removed</Uuid>
      ... many more attributes at this same level ...
    <User>
    <User>
      <Id>removed</Id>
      <Uuid>removed</Uuid>
      ... many more attributes at this same level ...
    <User>

Re: Is it possible to ingest XML?

smoir_splunk — Wed, 30 Sep 2020 03:01:32 GMT

This is not going to be props for your specific case (assuming you want each one of those users to be a separate event) but a similar example for props and transforms that I have to extract tracks from an itunes library XML file.
They're also visible here:
https://github.com/smoreface/music_app_for_splunk/blob/master/default/transforms.conf
https://github.com/smoreface/music_app_for_splunk/blob/master/default/props.conf

transforms stanza:

[itunes_xml]
CLEAN_KEYS = true
FORMAT = $1::$2
REGEX = <key>([^<]+)</key><[^>]+>([^<]+)</

props stanza:
[itunes_xml]
FIELDALIAS-iTunes_xml_Normie = Album AS album Artist AS artist Date_Added AS date_added Name AS track_name Play_Count AS play_count Play_Date AS last_played Play_Date_UTC AS last_played_utc Rating AS rating Release_Date AS release_date Size AS file_size Total_Time AS track_length Track_Number AS track_number
SEDCMD-xml&to& = s/&/&/g

Example XML being parsed:

<dict>
            <key>Track ID</key><integer>10815</integer>
            <key>Size</key><integer>4338490</integer>
            <key>Total Time</key><integer>216816</integer>
            <key>Track Number</key><integer>1</integer>
            <key>Track Count</key><integer>10</integer>
            <key>Year</key><integer>2004</integer>
            <key>Date Modified</key><date>2007-01-20T22:07:34Z</date>
            <key>Date Added</key><date>2008-07-27T03:52:43Z</date>
            <key>Bit Rate</key><integer>160</integer>
            <key>Sample Rate</key><integer>44100</integer>
            <key>Play Count</key><integer>1</integer>
            <key>Play Date</key><integer>3319660819</integer>
            <key>Play Date UTC</key><date>2009-03-12T07:00:19Z</date>
            <key>Skip Count</key><integer>1</integer>
            <key>Skip Date</key><date>2010-06-14T22:40:10Z</date>
            <key>Persistent ID</key><string>36990211F06BD125</string>
            <key>Track Type</key><string>File</string>
            <key>File Folder Count</key><integer>5</integer>
            <key>Library Folder Count</key><integer>1</integer>
            <key>Name</key><string>Cry</string>
            <key>Artist</key><string>Sirens</string>
            <key>Album</key><string>Tied To The Mast</string>
            <key>Genre</key><string>Pop</string>
            <key>Kind</key><string>MPEG audio file</string>
            <key>Location</key><string>file:///Users/user/Music/iTunes/iTunes%20Music/Music/Sirens/Tied%20To%20The%20Mast/01%20Cry.mp3</string>
        </dict>
        <key>10817</key>
        <dict>
            <key>Track ID</key><integer>10817</integer>
            <key>Size</key><integer>4082943</integer>
            <key>Total Time</key><integer>254093</integer>
            <key>Track Number</key><integer>1</integer>
            <key>Track Count</key><integer>2</integer>
            <key>Date Modified</key><date>2008-01-15T02:13:52Z</date>
            <key>Date Added</key><date>2008-07-27T03:52:43Z</date>
            <key>Bit Rate</key><integer>128</integer>
            <key>Sample Rate</key><integer>44100</integer>
            <key>Play Count</key><integer>19</integer>
            <key>Play Date</key><integer>3441386101</integer>
            <key>Play Date UTC</key><date>2013-01-19T04:35:01Z</date>
            <key>Skip Count</key><integer>1</integer>
            <key>Skip Date</key><date>2009-02-10T22:07:13Z</date>
            <key>Rating</key><integer>40</integer>
            <key>Album Rating</key><integer>20</integer>
            <key>Album Rating Computed</key><true/>
            <key>Persistent ID</key><string>36990211F06BD130</string>
            <key>Track Type</key><string>File</string>
            <key>File Folder Count</key><integer>5</integer>
            <key>Library Folder Count</key><integer>1</integer>
            <key>Name</key><string>Gone</string>
            <key>Artist</key><string>Straight No Chaser</string>
            <key>Kind</key><string>AAC audio file</string>
            <key>Location</key><string>file:///Users/user/Music/iTunes/iTunes%20Music/Music/Straight%20No%20Chaser/Unknown%20Album/01%20Gone.m4a</string>
        </dict>

Hope this helps!

Re: Is it possible to ingest XML?

woodcock — Wed, 20 Nov 2019 23:36:31 GMT

Just bring it in and set KV_MODE = xml in props.conf for your sourcetype on your Search Head(s).

topic Is it possible to ingest XML? in Getting Data In

Is it possible to ingest XML?

Re: Is it possible to ingest XML?

Re: Is it possible to ingest XML?