Multivalue xml field extraction

[jsanjeb]

Hi Splunkers,

Part of the incoming xml data looks like this,

<metaDataSet>
            <metaData key="DocName">mm1266505-20180330195251.txt</metaData>
            <metaData key="TARGET">Brazil BES</metaData>
            <metaData key="com.ibm.wmqfte.OriginatingUser">eimb@CORPWEB</metaData>
            <metaData key="com.ibm.wmqfte.OriginatingHost">B051S317.jci.com</metaData>
            <metaData key="com.ibm.wmqfte.TransferId">414d512050445053514d513031202020770fbe5aacf88f20</metaData>
            <metaData key="com.ibm.wmqfte.Priority">0</metaData>
        </metaDataSet>

I need extract the field key which is multi-valued. Current regex will extract only the first matching pattern. Please help with the regex.

Thanks

[xpac · Accepted solution]

Hey,

a proper regex would be \skey="([^"]+)".
Put it in your transforms, set MV_ADD = true, and you should be good.

However, this might be helpful, too: <metaData\s+key="([^"]+)">((?!<\/metaData>).+)<\/metaData>
You could then set FORMAT = $1::$2 so you get fields corresponding with the key names and their proper values.

You could also do both, like this:

props.conf
[your-sourcetype]

REPORT-metadata-fields = metadata-keys-mv-field, metadata-key-value-fields

transforms.conf

[metadata-keys-mv-field]
REGEX = <metaData\s+key="(?<metadata_keys>[^"]+)"
MV_ADD = true

[metadata-key-value-fields]
REGEX = <metaData\s+key="([^"]+)">((?!<\/metaData>).+)<\/metaData>
FORMAT = $1::$2
MV_ADD = true

Hope that helps - if it does I'd be happy if you would upvote/accept this answer, so others could profit from it. 🙂

[jsanjeb]

Hi xpac, thanks for response. I am working on Splunk cloud. Will i be able to add these extraction through UI?

[xpac]

I've no experience on Splunk Cloud, but on a on premise installation, you would have to do it via config files - no way to do this via the GUI. So unless Splunk Cloud doesn't offer something special for this case, I guess your way is through support then.

[jsanjeb]

I tried in cloud, field extraction is working perfectly . Thanks !! but looks like the second transform which maps values is not working. When I search a metadata_key with its values in a query it is not giving me results.

[xpac]

Did you add it via GUI? The FORMAT = $1::$2 is essential, else it will most likely not return anything.
I tried that regex here with your sample data, so at least the regex should be fine:
https://regex101.com/r/5JcfIv/1

[jsanjeb]

Yes I have added it via GUI. I have added FORMAT = $1::$2 as well. Regex is perfectly fine.

[xpac]

Ah, didn't know it was possible, rarely use the GUI. I fear without actual access troubleshooting this is difficult - maybe you can find any errors in index=_internal?

[jsanjeb]

Ok. Thanks !!