Define custom sourcetype XML

raymondc — Tue, 08 Mar 2016 11:56:20 GMT

I'm trying to define a custom sourcetype. I have one file with multiple XML files.

For example MyFile.xml:

<?xml version = '1.0' encoding = 'UTF-8' standalone = 'yes'?><DATA>
   <Time>2016-02-12T00:00:00.211Z</Time>
   <Item>
      <ID>1545454</ID>
      <VAR1>897654564dDJUHFKHJHEU</VAR1>
   </Item>
   <Check>OK</Check>
</DATA>
<?xml version = '1.0' encoding = 'UTF-8' standalone = 'yes'?><DATA>
   <Time>2016-02-12T00:00:00.211Z</Time>
   <Item>
      <ID>456849836848</ID>
      <VAR5>78964DFDFli_DFDFD_DFDF</VAR5>
   </Item>
   <Check>FAILD</Check>
</DATA>
<?xml version = '1.0' encoding = 'UTF-8'?><LOG>
   <Send>FKK_64646464</Send>
   <TimestampSend>2016-02-08T04:44:53.417Z</TimestampSend>
   <By>MFF_5687654</By>
   <MessageId>Title Test</MessageId>
   <Message>
      <Resp>
         <EventTime>2016-02-08T04:44:53.418Z</EventTime>
         <Info>
            <Item>
               <Id>INFO_222</Id>
            </Item>
            <Description>
               <Id>BCC_456</Id>
               <ByID>45</ByID>
            </Description>
         </Info>
         <Status>404</Status>
      </Resp>
   </Message>
</LOG>
<?xml version = '1.0' encoding = 'UTF-8' standalone = 'yes'?><DATA>
   <Time>2016-02-12T00:00:00.211Z</Time>
   <Item>
      <ID>45454545</ID>
      <VAR88>LJDKI_DFDFDF_DFDFDF_EJRHUHEJK</VAR88>
   </Item>
   <Check>WARNING</Check>
</DATA>
<?xml version = '1.0' encoding = 'UTF-8'?><LOG>
   <Send>FKK_64646465</Send>
   <TimestampSend>2016-02-08T04:48:53.417Z</TimestampSend>
   <By>MFF_5687655</By>
   <MessageId>Title Test</MessageId>
   <Message>
      <Resp>
         <EventTime>2016-02-08T04:48:53.418Z</EventTime>
         <Info>
            <Item>
               <Id>INFO_223</Id>
            </Item>
            <Description>
               <Id>BCC_457</Id>
               <ByID>46</ByID>
            </Description>
         </Info>
         <Status>404</Status>
      </Resp>
   </Message>
</LOG>

My props.conf

[mysourcetype]
 DATETIME_CONFIG = CURRENT
 KV_MODE = xml
 LINE_BREAKER = (<?xml)
 NO_BINARY_CHECK = 1
 SHOULD_LINEMERGE = false
 TRUNCATE = 0
 pulldown_type = 1

How can I set the LINE_BREAKER by "<?xml version = '1.0' encoding = 'UTF-8' standalone = 'yes'?>"
Is there a way te set the DATETIME to or with regex?
What did I do wrong?

Re: Define custom sourcetype XML

somesoni2 — Tue, 08 Mar 2016 18:25:41 GMT

Give this a try (props.conf on Indexer/Heavy forwarder)

[yoursourcetype]
SHOULD_LINEMERGE=true
LINE_BREAKER=(\<\?xml[^\?]+\?\>)
TIME_PREFIX=Time\s*\>
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%N

topic Re: Define custom sourcetype XML in Getting Data In

Define custom sourcetype XML

Re: Define custom sourcetype XML