topic Re: How to index a text file in the xml format? in Splunk Search

How to index a text file in the xml format?

madhanbaskar — Fri, 11 Aug 2017 08:12:00 GMT

/getClientProfileV1Request></SOAP-ENV:Body></SOAP-ENV:Envelope></soap-env:Body>-- HTTP Header values -<tp:headers xsi:type="http:HttpRequestHeaders" xmlns:http="http://www.bea.com/wli/sb/transports/http" xmlns:tp="http://www.bea.com/wli/sb/transports" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><tp:user-header name="Authorization" value="Basic cHJvZmlsZXVzZXI6cHJvZmlsM3VzM3I="/><tp:user-header name="EData" value="19=1.2,15=RS-Ro,16=Dn,13=getCtRequest,11=REQ,2=getes,10=3xbhkxv0p9,7=Assigned,6=PCK069,5=getAssignedRoles,4=com,9=roscha3c,8=UNKNOWN,1.2=APP10865:3xbhkxw1kw:3xbhkxxefh,1.1=APP09470::3xbhkxv28o"/><tp:user-header name="Syste" value="RO"/><http:Accept>application/xml</http:Accept><http:Content-Type>text/xml</http:Content-Type></tp:headers>

Above is the log in text format inside a text file. I need to index the same in my indexer in XML format itself. what would be the easy way to read the text file and convert it to an XML file so that I can view the logs in xml format in splunk GUI?

Re: How to index a text file in the xml format?

niketn — Fri, 11 Aug 2017 19:37:07 GMT

@madhanbaskar, you can either define KV_MODE=xml in props.conf for your input for Splunk to perform field discovery for XML data.

Other option would be to use spath to traverse the xml data once it is indexed.

Refer to documentation: https://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Automatickey-valuefieldextractionsatsearch-time#Automatic_key-value_field_extraction_and_the_sequence_of_search_operations

https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Spath

Re: How to index a text file in the xml format?

madhanbaskar — Wed, 16 Aug 2017 06:46:24 GMT

@niketnilay, Thanks

Will the above really help?

My problem is, in the client server the logs are stored in text format(.txt) but I need to forward that log to splunk server and index those logs in XML format(.xml).

What would be the easy way to do that?

I'm not sure of how the Spath or KV_mode help in this transformation case?

Re: How to index a text file in the xml format?

niketn — Wed, 16 Aug 2017 07:19:30 GMT

On a test machine or your local machine, you can take a sample log containing XML data and upload to Splunk (follow steps from the documentation: http://docs.splunk.com/Documentation/Splunk/latest/SearchTutorial/GetthetutorialdataintoSplunk)
Once you are in Sourcetype screen, from Advanced settings you can add KV_MODE=xml

One of the other things you can try out is upload the data as txt and just runspath command to extract fields as you need. https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Spath. PS: if your file has content other XML you can use rex command to extract xml data first, before applying spath.

Does the text file contain only XML or does it contain XML between some text data as delimiters? Would you be able to post the content of a sample file after masking/anonymizing sensitive data?

Re: How to index a text file in the xml format?

madhanbaskar — Wed, 16 Aug 2017 10:13:05 GMT

@niketnilay,

The difference is these 3 tags < , " , &gt; in the text file needs to be changed to < , " ,> respectively

If so then the file will be automatically converted to a xml file.... better?

Re: How to index a text file in the xml format?

niketn — Wed, 16 Aug 2017 10:44:30 GMT

Sorry I just noticed that you might not be interested in XML spath rather value containing within XML nodes as Key Value Pairs. Splunk would have been able to directly extract Key Value pairs. However, in this case I see value=key, rather than key=value. Splunk's default fields can not start with numbers, hence can not be all numbers also. Besides you have multiple fields with name. Only first one will get extracted by default.

So, in your case I would suggest to go with Regular Expression based Field Extraction:http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/ExtractfieldsinteractivelywithIFX

In the above sample data can you please let me know what are the fields you are interested in?

Re: How to index a text file in the xml format?

ddrillic — Wed, 16 Aug 2017 10:53:37 GMT

A similar case I had recently at How can we index an entire XML document as one event?

Re: How to index a text file in the xml format?

madhanbaskar — Wed, 16 Aug 2017 11:00:05 GMT

@niketnilay,

si=& quot ;http://www.w3.org/2001/XMLSchema-instance & quot;& gt ;& lt ;tp:headers

In the above sample i need to change the & quot ; & gt; & lt ;

to " , > , < respectively.

That should help me 😞

Re: How to index a text file in the xml format?

niketn — Wed, 16 Aug 2017 11:50:04 GMT

Can you re-post XML or edit your question with Code button (101010) here on Splunk Answers so that special characters are not escaped?

I think your original post does not have < " and > characters, rather have <, ", >. Please confirm if this is correct.

Re: How to index a text file in the xml format?

madhanbaskar — Thu, 17 Aug 2017 11:01:23 GMT

@niketnilay , Exactly!!

You are right. The original post doesnt have < > " rather it has lt,quot,gt ....

I need to know how to change that in splunk so that in the place of lt,quot and gt , < " > will get replaced??

Re: How to index a text file in the xml format?

niketn — Thu, 17 Aug 2017 11:50:25 GMT

Can you please try the following:

eval xml_data=replace(replace(replace(_raw,"&lt;","<"),"&gt;",">"),"&quot;","\"")
| table xml_data

Re: How to index a text file in the xml format?

madhanbaskar — Thu, 17 Aug 2017 15:55:37 GMT

@niketnilay ,

Thanks that actually helped 🙂

Re: How to index a text file in the xml format?

niketn — Thu, 17 Aug 2017 16:19:30 GMT

@madhanbaskar, I have converted the comment to answer. Please accept to mark this as answered.