Getting Data In

how to parse an xml file as 1 event

I have an XML file I want to bring into splunk as a single event. It is the equivalent of an Excel file. The props.conf is set as:


[excelxml]  
SHOULD_LINEMERGE = true 
TRUNCATE = 0 
MAX_EVENTS = 500000 
TIME_PREFIX = \ 
TIME_FORMAT = %d:%m:%Y:%h:%m:%s 

This does result in an event that matches the file, and it has the correct timestamp. But I also get another event with some of the row data in it.

The event I want shows all the data. The event I don't want starts at the first tag.

I suspect that some default xml processing is breaking out the row data, but I am not sure how to suppress this. Any hints appreciated.

Thanks

Tags (1)

Ultra Champion

I think you're going to want to use LINE_BREAKER to specify where the event should break. http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf and http://docs.splunk.com/Documentation/Splunk/latest/Data/Configureeventlinebreaking elaborates on its usage.

Would you review that and reply back with an attempted solution that defines the line break and if it works or not? We can review from there.

0 Karma

Communicator

this resolved.

0 Karma

Ultra Champion

Sounds like that worked. If so, you can click the link to accept this answer OR, if something else solved it, let us know what.

0 Karma

Ultra Champion

On the phone you mentioned this config was on the search head and the forwarder. Is it also on the indexer? That's where the event would get processed. This might help clarify: http://docs.splunk.com/Documentation/Splunk/latest/Admin/Configurationparametersandthedatapipeline#H...

If still no good, maybe post a sanitized example of the data so we can see how if the props are good.

0 Karma

Communicator

Hello Burch,

I must have said wrong on the call. The config is on the forwarder and on indexers.
Below is a sample. It has more lines similar. Around 8000. We want that whole 8000 lines as one event.

<?xml version="1.0" encoding="UTF-8" ?>
<?xml-stylesheet type='text/xsl' href='freb.xsl'?>
<!-- saved from url=(0014)about:internet -->
<failedRequest url="https://xxxxx.fmr.com:443/xxxxxx/xxxxxxx.aspx"
               siteId="21"
               appPoolId="xxx_xx"
               processId="xxxxx"
               verb="POST"
               remoteUserName=""
               userName=""
               tokenUserName="NT AUTHORITY\IUSR"
               authenticationType="anonymous"
               activityId="{00000000-0000-0000-D07B-0080010000DB}"
               failureReason="STATUS_CODE"
               statusCode="500"
               triggerStatusCode="500"
               timeTaken="765"
               xmlns:freb="http://schemas.microsoft.com/win/2006/06/iis/freb"
               >
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
 <System>
  <Provider Name="WWW Server" Guid="{3A2A4E84-4C21-4981-AE10-3FDA0D9B0F83}"/>
  <EventID>0</EventID>
  <Version>1</Version>
  <Level>5</Level>
  <Opcode>2</Opcode>
  <Keywords>0x100</Keywords>
  <TimeCreated SystemTime="2017-02-08T18:02:20.952Z"/>
  <Correlation ActivityID="{00000000-0000-0000-D07B-0080010000DB}"/>
  <Execution ProcessID="12812" ThreadID="16488"/>
  <Computer>xxxxxxx</Computer>
 </System>
 <EventData>
  <Data Name="ContextId">{00000000-0000-0000-D07B-0080010000DB}</Data>
  <Data Name="ModuleName">FailedRequestsTracingModule</Data>
  <Data Name="Notification">1</Data>
  <Data Name="fIsPostNotificationEvent">false</Data>
  <Data Name="NotificationStatus">0</Data>
 </EventData>
 <RenderingInfo Culture="en-US">
  <Opcode>NOTIFY_MODULE_END</Opcode>
  <Keywords>
   <Keyword>RequestNotifications</Keyword>
  </Keywords>
  <freb:Description Data="Notification">BEGIN_REQUEST</freb:Description>
  <freb:Description Data="NotificationStatus">NOTIFICATION_CONTINUE</freb:Description>
 </RenderingInfo>
 <ExtendedTracingInfo xmlns="http://schemas.microsoft.com/win/2004/08/events/trace">
  <EventGuid>{002E91E3-E7AE-44AB-8E07-99230FFA6ADE}</EventGuid>
 </ExtendedTracingInfo>
</Event>
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
 <System>
  <Provider Name="WWW Server" Guid="{3A2A4E84-4C21-4981-AE10-3FDA0D9B0F83}"/>
  <EventID>0</EventID>
  <Version>1</Version>
  <Level>5</Level>
  <Opcode>1</Opcode>
  <Keywords>0x100</Keywords>
  <TimeCreated SystemTime="2017-02-08T18:02:20.952Z"/>
  <Correlation ActivityID="{00000000-0000-0000-D07B-0080010000DB}"/>
  <Execution ProcessID="12812" ThreadID="16488"/>
  <Computer>xxxxxxx</Computer>
 </System>
 <EventData>
  <Data Name="ContextId">{00000000-0000-0000-D07B-0080010000DB}</Data>
  <Data Name="ModuleName">ConfigurationValidationModule</Data>
  <Data Name="Notification">1</Data>
  <Data Name="fIsPostNotification">false</Data>
 </EventData>
 <RenderingInfo Culture="en-US">
  <Opcode>NOTIFY_MODULE_START</Opcode>
  <Keywords>
   <Keyword>RequestNotifications</Keyword>
  </Keywords>
  <freb:Description Data="Notification">BEGIN_REQUEST</freb:Description>
 </RenderingInfo>
 <ExtendedTracingInfo xmlns="http://schemas.microsoft.com/win/2004/08/events/trace">
  <EventGuid>{002E91E3-E7AE-44AB-8E07-99230FFA6ADE}</EventGuid>
 </ExtendedTracingInfo>
</Event>
</failedRequest>
0 Karma

Ultra Champion

Just to be safe, I want to make sure you see that there's multiple stanzas of the same type (example: EventData). As such, you might need to play with the sourcetype to allow multivalue fields to retain all data.

0 Karma

Communicator

I tried to do this, but sill i am not getting it as one full event, i put below in my forwarder and index props.conf

[ name ]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
CHARSET=UTF-8
disabled=false
BREAK_ONLY_BEFORE=GobblyGook
MAX_EVENTS=100000
0 Karma

Ultra Champion

'GobblyGook' was meant to be text that you replace and not to be implemented literally.

0 Karma

SplunkTrust
SplunkTrust

Heh. It would work anyway, as long as the file never contains that literal.

0 Karma

Ultra Champion

ha ha ha ha

0 Karma

Motivator

I remember reading, that setting:

BREAK_ONLY_BEFORE = GobblyGook

in props.conf will make Splunk read the entire file since GobblyGook never appears in the source file. We use this for non-XML files I'm not sure if this works with XML Files.

Chris

Thanks Chris
I put GobblyGook in props.conf as suggested and now it is one event! As I wanted ...

I still wonder what was causing the extra event - I have seen reference to default XML file processing in other places and wonder if that kicked in as well - but that's maybe a question for another day.

BR
Steve

0 Karma
Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes
and swag!