topic Re: XML log source type in Getting Data In

XML log source type

cparham — Wed, 05 May 2010 01:05:48 GMT

I cannot find much helpful documentation on handling XML log files. This link seems to be on the right track but what happens if I have multiple log file formats with different element names: http://www.splunk.com/wiki/Deploy:HowToWorkWithXMLLogFiles

Am I supposed to see xml_file as a source file type? I don't see it in the list of source types.

Here's what I'm wanting to parse. Its an Oracle database xml audit log:

<?xml version="1.0" encoding="UTF-8"?>
<Audit xmlns="http://xmlns.oracle.com/oracleas/schema/dbserver_audittrail-10_2.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://xmlns.oracle.com/oracleas/schema/dbserver_audittrail-10_2.xsd">
<Version>10.2</Version>
<AuditRecord>
    <Audit_Type>1</Audit_Type>
    <Session_Id>735807</Session_Id>
    <StatementId>9</StatementId>
    <EntryId>1</EntryId>
    <Extended_Timestamp>2010-05-02T23:14:49.399000</Extended_Timestamp>
    <DB_User>FRAMEWK</DB_User>
    <Client_Id>framewk</Client_Id>
    <OS_User>jsoap</OS_User>
    <Userhost>BIOA171</Userhost>
    <OS_Process>1680:6008</OS_Process>
    <Terminal>unknown</Terminal>
    <Instance_Number>0</Instance_Number>
    <Object_Schema>FRAMEWK</Object_Schema>
    <Object_Name>APP_USER</Object_Name>
    <Action>3</Action>
    <Returncode>0</Returncode>
    <Scn>1098595781</Scn>
    <Sql_Text>SELECT ... FROM ...</Sql_Text>
</AuditRecord>
</Audit>

Re: XML log source type

John_Mark — Wed, 05 May 2010 01:53:05 GMT

I think you have to define it as a source type - and use the example above as a guide.

Re: XML log source type

Lowell — Wed, 05 May 2010 21:59:48 GMT

I have a number of oracle related sourcetypes defined in a custom app, but I haven't come across this one before. Here is an XML config that I've modified to match your given log file sample.

Entries for props.conf:

[source::...[/\\][Oo]racle...[/\\]logs?...[/\\]audit.xml]
sourcetype = oracle_audit_xml

[oracle_audit_xml]
TIME_PREFIX = <Extended_Timestamp>
MAX_TIMESTAMP_LOOKAHEAD = 200
BREAK_ONLY_BEFORE = ^<AuditRecord>
BREAK_ONLY_BEFORE_DATE = False
SHOULD_LINEMERGE = True

Notes:

You will need to update the [source::] pattern based on the file names of these log files.
I'm assuming that if you have multiple log entries in a single file, then it's the <AuditRecord> will occur multiple times. Event breaking with XML is klugy, since XML was never meant for logs (IMHO), and it can't properly exist in fragments, so you'll probably end up with some extra junk before/after some of the very first or very last log entries.

If you provide a multiple-event example, someone could probably give you a better starting point.

Re: XML log source type

bfaber — Thu, 06 May 2010 18:37:02 GMT

Lowell's answer here looks pretty sane. Remember that once the breaking is done, you can pipe results to xmlkv for some pretty nifty automatic field extraction.

Re: XML log source type

cparham — Thu, 05 Aug 2010 03:56:04 GMT

Any suggestions on how to remove line breaks found inside elements .... Many of the SQL statements inside tags have hard line breaks often at very undesirable places.

Re: XML log source type

Lowell — Thu, 05 Aug 2010 20:29:36 GMT

You can do this at search time by doing something like this: sourcetype=oracle_audit_xml | xmlkv | regex field=Sql_Text mode=sed "s/[\r\n]+/ /g" Basically we are just replacing any end-of-line characters with a single space. Another approach would be to modify the raw event at index time, but that would be more difficult and I wouldn't recommend it unless you have some really good reasons.

Re: XML log source type

Lowell — Thu, 05 Aug 2010 21:09:07 GMT

Whoops, that should be "rex" not "regex" in the example command above.