topic Re: Multivalue xml field extraction in Splunk Search

Multivalue xml field extraction

jsanjeb — Mon, 30 Apr 2018 08:36:01 GMT

Hi Splunkers,

Part of the incoming xml data looks like this,

<metaDataSet>
            <metaData key="DocName">mm1266505-20180330195251.txt</metaData>
            <metaData key="TARGET">Brazil BES</metaData>
            <metaData key="com.ibm.wmqfte.OriginatingUser">eimb@CORPWEB</metaData>
            <metaData key="com.ibm.wmqfte.OriginatingHost">B051S317.jci.com</metaData>
            <metaData key="com.ibm.wmqfte.TransferId">414d512050445053514d513031202020770fbe5aacf88f20</metaData>
            <metaData key="com.ibm.wmqfte.Priority">0</metaData>
        </metaDataSet>

I need extract the field key which is multi-valued. Current regex will extract only the first matching pattern. Please help with the regex.

Thanks

Re: Multivalue xml field extraction

xpac — Mon, 30 Apr 2018 08:59:34 GMT

Hey,

a proper regex would be \skey="([^"]+)".
Put it in your transforms, set MV_ADD = true, and you should be good.

However, this might be helpful, too: <metaData\s+key="([^"]+)">((?!<\/metaData>).+)<\/metaData>
You could then set FORMAT = $1::$2 so you get fields corresponding with the key names and their proper values.

You could also do both, like this:

props.conf
[your-sourcetype]

REPORT-metadata-fields = metadata-keys-mv-field, metadata-key-value-fields

transforms.conf

[metadata-keys-mv-field]
REGEX = <metaData\s+key="(?<metadata_keys>[^"]+)"
MV_ADD = true

[metadata-key-value-fields]
REGEX = <metaData\s+key="([^"]+)">((?!<\/metaData>).+)<\/metaData>
FORMAT = $1::$2
MV_ADD = true

Hope that helps - if it does I'd be happy if you would upvote/accept this answer, so others could profit from it. 🙂

Re: Multivalue xml field extraction

jsanjeb — Mon, 30 Apr 2018 09:09:35 GMT

Hi xpac, thanks for response. I am working on Splunk cloud. Will i be able to add these extraction through UI?

Re: Multivalue xml field extraction

xpac — Mon, 30 Apr 2018 09:14:17 GMT

I've no experience on Splunk Cloud, but on a on premise installation, you would have to do it via config files - no way to do this via the GUI. So unless Splunk Cloud doesn't offer something special for this case, I guess your way is through support then.

Re: Multivalue xml field extraction

jsanjeb — Mon, 30 Apr 2018 10:21:55 GMT

I tried in cloud, field extraction is working perfectly . Thanks !! but looks like the second transform which maps values is not working. When I search a metadata_key with its values in a query it is not giving me results.

Re: Multivalue xml field extraction

xpac — Mon, 30 Apr 2018 10:25:53 GMT

Did you add it via GUI? The FORMAT = $1::$2 is essential, else it will most likely not return anything.
I tried that regex here with your sample data, so at least the regex should be fine:
https://regex101.com/r/5JcfIv/1

Re: Multivalue xml field extraction

jsanjeb — Mon, 30 Apr 2018 10:30:03 GMT

Yes I have added it via GUI. I have added FORMAT = $1::$2 as well. Regex is perfectly fine.

Re: Multivalue xml field extraction

xpac — Mon, 30 Apr 2018 10:34:42 GMT

Ah, didn't know it was possible, rarely use the GUI. I fear without actual access troubleshooting this is difficult - maybe you can find any errors in index=_internal?

Re: Multivalue xml field extraction

jsanjeb — Mon, 30 Apr 2018 11:21:17 GMT

Ok. Thanks !!