Splunk Search

Multivalue xml field extraction

jsanjeb
Explorer

Hi Splunkers,

Part of the incoming xml data looks like this,

<metaDataSet>
            <metaData key="DocName">mm1266505-20180330195251.txt</metaData>
            <metaData key="TARGET">Brazil BES</metaData>
            <metaData key="com.ibm.wmqfte.OriginatingUser">eimb@CORPWEB</metaData>
            <metaData key="com.ibm.wmqfte.OriginatingHost">B051S317.jci.com</metaData>
            <metaData key="com.ibm.wmqfte.TransferId">414d512050445053514d513031202020770fbe5aacf88f20</metaData>
            <metaData key="com.ibm.wmqfte.Priority">0</metaData>
        </metaDataSet>

I need extract the field key which is multi-valued. Current regex will extract only the first matching pattern. Please help with the regex.

Thanks

Tags (1)
0 Karma
1 Solution

xpac
SplunkTrust
SplunkTrust

Hey,

a proper regex would be \skey="([^"]+)".
Put it in your transforms, set MV_ADD = true, and you should be good.

However, this might be helpful, too: <metaData\s+key="([^"]+)">((?!<\/metaData>).+)<\/metaData>
You could then set FORMAT = $1::$2 so you get fields corresponding with the key names and their proper values.

You could also do both, like this:

props.conf
[your-sourcetype]

REPORT-metadata-fields = metadata-keys-mv-field, metadata-key-value-fields

transforms.conf

[metadata-keys-mv-field]
REGEX = <metaData\s+key="(?<metadata_keys>[^"]+)"
MV_ADD = true

[metadata-key-value-fields]
REGEX = <metaData\s+key="([^"]+)">((?!<\/metaData>).+)<\/metaData>
FORMAT = $1::$2
MV_ADD = true

Hope that helps - if it does I'd be happy if you would upvote/accept this answer, so others could profit from it. 🙂

View solution in original post

0 Karma

xpac
SplunkTrust
SplunkTrust

Hey,

a proper regex would be \skey="([^"]+)".
Put it in your transforms, set MV_ADD = true, and you should be good.

However, this might be helpful, too: <metaData\s+key="([^"]+)">((?!<\/metaData>).+)<\/metaData>
You could then set FORMAT = $1::$2 so you get fields corresponding with the key names and their proper values.

You could also do both, like this:

props.conf
[your-sourcetype]

REPORT-metadata-fields = metadata-keys-mv-field, metadata-key-value-fields

transforms.conf

[metadata-keys-mv-field]
REGEX = <metaData\s+key="(?<metadata_keys>[^"]+)"
MV_ADD = true

[metadata-key-value-fields]
REGEX = <metaData\s+key="([^"]+)">((?!<\/metaData>).+)<\/metaData>
FORMAT = $1::$2
MV_ADD = true

Hope that helps - if it does I'd be happy if you would upvote/accept this answer, so others could profit from it. 🙂

0 Karma

jsanjeb
Explorer

Hi xpac, thanks for response. I am working on Splunk cloud. Will i be able to add these extraction through UI?

0 Karma

xpac
SplunkTrust
SplunkTrust

I've no experience on Splunk Cloud, but on a on premise installation, you would have to do it via config files - no way to do this via the GUI. So unless Splunk Cloud doesn't offer something special for this case, I guess your way is through support then.

0 Karma

jsanjeb
Explorer

I tried in cloud, field extraction is working perfectly . Thanks !! but looks like the second transform which maps values is not working. When I search a metadata_key with its values in a query it is not giving me results.

0 Karma

xpac
SplunkTrust
SplunkTrust

Did you add it via GUI? The FORMAT = $1::$2 is essential, else it will most likely not return anything.
I tried that regex here with your sample data, so at least the regex should be fine:
https://regex101.com/r/5JcfIv/1

0 Karma

jsanjeb
Explorer

Yes I have added it via GUI. I have added FORMAT = $1::$2 as well. Regex is perfectly fine.

xpac
SplunkTrust
SplunkTrust

Ah, didn't know it was possible, rarely use the GUI. I fear without actual access troubleshooting this is difficult - maybe you can find any errors in index=_internal?

0 Karma

jsanjeb
Explorer

Ok. Thanks !!

0 Karma
Get Updates on the Splunk Community!

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...