topic Re: How to break the XML file into multiple events from search head? in Splunk Search

How to break the XML file into multiple events from search head?

nasrinmulani — Fri, 27 Jul 2018 06:36:08 GMT

Hi All,

I have indexed the XML file without breaking it into events, I need to break the events using on tag. Hence can i achieve this using the search head?
Example

I need to break the events by tag, so that i can fetch the details in each tag as each row.

Thanks in advance!

Re: How to break the XML file into multiple events from search head?

niketn — Fri, 27 Jul 2018 06:51:43 GMT

@nasrinmulani I think you have missed the example. Please mock/anonymize any sensitive data before posting.

Re: How to break the XML file into multiple events from search head?

nasrinmulani — Fri, 27 Jul 2018 11:08:29 GMT

Thanks Niket,

I don't it's getting replacing with blank place.
But Example is -
t1 tag start
t1 tag end
t2 tag start
t2 tag end
t2 tag start
t2 tag end
t2 tag start
t2 tag end
t3 tag start
t3 tag end

Re: How to break the XML file into multiple events from search head?

nasrinmulani — Fri, 27 Jul 2018 11:09:53 GMT

I don't know it's getting replaced by blank place

Re: How to break the XML file into multiple events from search head?

jplumsdaine22 — Fri, 27 Jul 2018 14:05:07 GMT

In the text editor for putting your question theres a little picture of some 1s and zeroes. It will open a code box you can paste your xml in there

Re: How to break the XML file into multiple events from search head?

nasrinmulani — Tue, 31 Jul 2018 05:58:32 GMT

<Head>
<Detail>
<id>123</id>
<Name>x</Name>
</Detail>
</Head>
<Head>
<Detail>
<id>1234</id>
<Name>y</Name>
</Detail>
</Head>
<Head>
<Detail>
<id>12347</id>
<Name>y</Name>
</Detail>
</Head>

Re: How to break the XML file into multiple events from search head?

nasrinmulani — Tue, 31 Jul 2018 06:00:54 GMT

I want to break this into multiple events, it is not a single event.
I want break it through tag but in search head.

Is there any SPL command to break the events from search head?

Re: How to break the XML file into multiple events from search head?

niketn — Tue, 31 Jul 2018 06:15:07 GMT

@nasrinmulani, please try the following two options with run anywhere examples:

Option 1: Using spath

| makeresults
| eval _raw="<Head>
 <Detail>
 <id>123</id>
 <Name>x</Name>
 </Detail>
 </Head>
 <Head>
 <Detail>
 <id>1234</id>
 <Name>y</Name>
 </Detail>
 </Head>
 <Head>
 <Detail>
 <id>12347</id>
 <Name>y</Name>
 </Detail>
 </Head>"
| spath
| fields - _*
| rename "Head.Detail.*" as "*"
| eval data=mvzip(id,Name)
| fields data
| mvexpand data
| makemv data delim=","
| eval id=mvindex(data,0), Name=mvindex(data,1)
| fields - data

Option 2: Using rex command

| makeresults
| eval _raw="<Head>
 <Detail>
 <id>123</id>
 <Name>x</Name>
 </Detail>
 </Head>
 <Head>
 <Detail>
 <id>1234</id>
 <Name>y</Name>
 </Detail>
 </Head>
 <Head>
 <Detail>
 <id>12347</id>
 <Name>y</Name>
 </Detail>
 </Head>"
| rex "(?ms)(?<Detail>\<Detail\>.*?\<\/Detail\>)" max_match=0
| fields - _*
| mvexpand Detail
| rename Detail as _raw
| spath

PS: First two pipes i.e. | makeresults and | eval _raw .... generate sample event as per the question. You should use your own base search instead of these to test with your data.

Re: How to break the XML file into multiple events from search head?

jplumsdaine22 — Tue, 31 Jul 2018 08:16:38 GMT

As an extra note, unless your XML will always have the same structure, use | spath (IE Option 1) instead of rex to extract the fields.

Re: How to break the XML file into multiple events from search head?

efavreau — Tue, 31 Jul 2018 12:52:58 GMT

Look into using spath within an eval statement, if you find the spath command is slow. We found that rewriting spath to using spath within an eval, and the occasional rex, is much faster. Use the job inspector to measure your results.

Note it was a lot of work to rewrite the command into eval and rex, but for as often as we needed it, for the amount of data we needed it for, it was worth the effort. YMMV.

The Doc for Eval: http://docs.splunk.com/Documentation/Splunk/7.1.2/SearchReference/CommonEvalFunctions

From that page:

spath(X,Y) Extracts a value from a
structured data type (XML or JSON) in
X based on a location path in Y.

Re: How to break the XML file into multiple events from search head?

nasrinmulani — Tue, 29 Sep 2020 20:47:13 GMT

Thanks!! It was helpful for me, but now i am thinking to break it via props.conf itself.
I got one issue that my log events are breaking in between, may it's because of larger size of records so it's truncating it.

Any one know about the breaking of file. I am trying with
BREAK_ONLY_BEFORE=
MUST_BREAK_AFTER=<\/Detail>

let's see if it works.

Re: How to break the XML file into multiple events from search head?

niketn — Sat, 04 Aug 2018 15:10:15 GMT

Change the TRUNCATE limit as per your data refer more options for Event Breaking: https://docs.splunk.com/Documentation/Splunk/latest/Data/Configureeventlinebreaking

Test with some of the xml data for which it is not working in Preview mode with sample data.