topic Re: XML Extraction in Dashboards & Visualizations

XML Extraction

jedatt01 — Tue, 29 Apr 2014 13:20:22 GMT

I have a datasource that reads in events in XML format. Could someone please help me build a props.conf that will extract all fields and show the events in treeview. Sample event below

Mon Apr 28 16:45:57 EDT 2014 name="TOPIC_msg_received" event_id="ID:404040" msg_dest="SplunkTopic" msg_body="<?xml version="1.0" encoding="utf-8"?><ELLogInputLayout xmlns="http://www.test.com/1"><ELLogInputMessage>    <Header>      <LogEventTypeCode>ERROR</LogEventTypeCode>      <LogSeverityCode>CRITICAL</LogSeverityCode>      <LogEventDateTime>9999-12-31T23:59:59.9999999-05:00</LogEventDateTime>    </Header>    <SourceInformation>      <EAPMId>2</EAPMId>      <HostMachineName>HostMachineName3</HostMachineName>      <HostEnvironmentName>HostEnvironmentName3</HostEnvironmentName>      <ComponentId>ComponentId3</ComponentId>      <ComponentName>ComponentName3</ComponentName>      <ApplicationEventCorrelationId>ApplicationEventCorrelationId3</ApplicationEventCorrelationId>      <UserId>UserId3</UserId>      <UserSrc>UserSrc3</UserSrc>      <BusinessDomainId>BusinessDomainId3</BusinessDomainId>      <BusinessDomainName>BusinessDomainName3</BusinessDomainName>    </SourceInformation>    <ErrorInformation>      <ErrorCode>ErrorCode3</ErrorCode>      <ErrorDescription>ErrorDescription3</ErrorDescription><DetailedErrorInformation>DetailedErrorInformation3</DetailedErrorInformation>    </ErrorInformation>    <DetailedLogInformation>anyType</DetailedLogInformation>   </ELLogInputMessage></ELLogInputLayout>"

Re: XML Extraction

gfuente — Tue, 29 Apr 2014 14:12:36 GMT

Hello

Have your tried KV_MODE = XML in props.conf ??

From docs:

KV_MODE = [none|auto|multi|json|xml]
* Used for search-time field extractions only.
* Specifies the field/value extraction mode for the data.
* Set KV_MODE to one of the following:
        * none: if you want no field/value extraction to take place.
        * auto: extracts field/value pairs separated by equal signs.
        * multi: invokes the multikv search command to expand a tabular event into multiple events.
    * xml : automatically extracts fields from XML data.
    * json: automatically extracts fields from JSON data.
* Setting to 'none' can ensure that one or more user-created regexes are not overridden by
  automatic field/value extraction for a particular host, source, or source type, and also
  increases search performance.
* Defaults to auto.
* The 'xml' and 'json' modes will not extract any fields when used on data that isn't of the correct format (JSON or XML).

Re: XML Extraction

somesoni2 — Tue, 29 Apr 2014 15:00:32 GMT

This may not work as the event is not pure xml (its a combination of key value pair with embedded xml). You might have to extract all xml fields using field extractor.

Re: XML Extraction

jedatt01 — Tue, 29 Apr 2014 15:04:54 GMT

Yes, I tried using KV_MODE = XML but it is not picking up. Is the only way to do individual field extraction?

Re: XML Extraction

somesoni2 — Tue, 29 Apr 2014 15:18:01 GMT

Try this.

props.conf

[yourSourceType]
NO_BINARY_CHECK = 1
TIME_FORMAT = %a %b %d %H:%M:%S %T %Y
pulldown_type = 1
REPORT-xmlkv = xmlkv-alternative

transforms.conf

[xmlkv-alternative]
REGEX = <([^\s\>]*)[^\>]*\>([^<]*)\<\/\1\>
FORMAT = $1::$2

Re: XML Extraction

jedatt01 — Tue, 29 Apr 2014 16:45:11 GMT

That extracted all the fields, thanks! Wish there was a way to get a pull down view but this will do the trick if that's not possible.

Re: XML Extraction

rahulroy_splunk — Tue, 29 Apr 2014 21:02:25 GMT

Could you be more specific what you mean by pull down view?

Re: XML Extraction

jedatt01 — Tue, 06 May 2014 14:42:37 GMT

when you have an input that is true json or xml you can have the view of the events in a tree structure where you hit the + sign to see nested information.

Re: XML Extraction

landen99 — Wed, 05 Oct 2016 13:48:32 GMT

I like this solution using transforms.conf

[views_std]
MV_ADD = 1
REGEX = \<(\w+[^\n\/\>]+)\/?\>([^\<\n][^\<]*)\<
FORMAT = $1::$2
CLEAN_KEYS = true

[views_param]
MV_ADD = 1
REGEX = \<(\w+ [^\n\/\>]+)\/?\>
FORMAT = param::$1
CLEAN_KEYS = true

[views_option]
MV_ADD = 1
SOURCE_KEY = param
REGEX = (\w+(?: \w+)*)="(?!host|source|sourcetype|index|splunk_server)(\w+)"
FORMAT = $1::$2
CLEAN_KEYS = true

Re: XML Extraction

poddraj — Mon, 25 Nov 2019 10:04:25 GMT

Hi,

I have a .log file which has mix of system logs and in between request & response XMLs of actual transactions going within my application. I only want those XML transactions only to be shown in splunk and do not want to see the other logging information. Is there a way to achieve this?