Single Line XML Log File

[jheilman]

We have a system called Metaviewer that produces XML log files. I've read several answers about indexing and trapping events from an XML file. The problem in this case is that all of the events are combined into a single line. The only time a line break even enters the picture is when one of the <SQL> elements in the XML event contains a line break. Is there a way to index multiple events from a single line?

A stripped down sample log event from Metaviewer might look like this...

<?xml version="1.0" encoding="UTF-8"?><!--Log file generated byMetafile Event Logging Server--><EventLog xmlns="http://www.metafile.com/ns/eventlog/10" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><CreateDate>6/17/2010</CreateDate><CreateTime>13:20:00</CreateTime><Logger>servername</Logger><Events><Event><EventID>256</EventID><Description>Modify User</Description><Category>Audit</Category><Source>MetaViewer Enterprise</Source><SubSource>MVENT</SubSource><UserName>DOMAIN\user</UserName><UserID>562</UserID><Computer>servername</Computer><Date>06/17/2010</Date><Time>13:24:33</Time><ObjectType>User</ObjectType><Details><SQL>UPDATE UserInfo SET CMAnnTlbx = 0,CMArchMaint = 0,CMArchSearch = 0,CMDocClass = 0,CMDocForms = 0,CMMetaViews = 0,CMMoveStrg = 0,CMPgStore = 0,CMPrefetch = 0,CMPurgeDoc = 0,CMQueries = 0,CMShortcuts = 0,CMSysInfo = 0,CMTasks = 0,CMUsers = 0,CMWorkGrps = 0,GroupID = 36,Notes = NULL,SIDType = 0,SIDValue = NULL,SigCode = 0,Attribs = &apos;&lt;Variables&gt;&lt;/Variables&gt;&apos;,InactivityTO = 0,NameFirst = NULL,NameLast = NULL,NameUser = &apos;UserName&apos; WHERE UserID =575</SQL><UserID>575</UserID><UserName>UserName</UserName></Details></Event></Events></EventLog>

In a full log, multiple <Event> elements would be slapped together on the same line. The <SQL> element may contain line breaks, but that's the only time you'll ever see them. So the challenge is to index each of the <Event>s that exist on the same line, while still trapping multiple lines when necessary.

I can't even seem to get Splunk to index multiple events on a single line. Here's what I'm trying right now...

[metaviewer_audit_xml]
TIME_PREFIX = <Date>
MAX_TIMESTAMP_LOOKAHEAD = 200
BREAK_ONLY_BEFORE = <Event>
BREAK_ONLY_BEFORE_DATE = False
MUST_BREAK_AFTER = </Event>
SHOULD_LINEMERGE = True

I feel like I know why this doesn't work, but I haven't found another way of doing it. Any suggestions?

[southeringtonp · Accepted solution]

LINEMERGE and related settings by nature operate on single lines and try to reassemble them into multi-line events. When all your data is on one line, the assumption is that it's all part of the same event.

Instead, take a look at LINE_BREAKER, which will allow you to break events up before line merging takes place. As a bonus, it's generally faster since you can turn merging off.

Fair warning -- I haven't tested this regex -- but it should give you the general idea:

[metaviewer_audit_xml]
LINE_BREAKER = [\>\s]((?=\<Event\>))
SHOULD_LINEMERGE = False
TIME_PREFIX = <Date>
MAX_TIMESTAMP_LOOKAHEAD = 350

You can find a little more information here:
     http://www.splunk.com/base/Documentation/4.1.6/Admin/Indexmulti-lineevents

[jheilman]

This worked perfectly. Thank you!