XML log source type

[cparham]

I cannot find much helpful documentation on handling XML log files. This link seems to be on the right track but what happens if I have multiple log file formats with different element names: http://www.splunk.com/wiki/Deploy:HowToWorkWithXMLLogFiles

Am I supposed to see xml_file as a source file type? I don't see it in the list of source types.

Here's what I'm wanting to parse. Its an Oracle database xml audit log:

<?xml version="1.0" encoding="UTF-8"?>
<Audit xmlns="http://xmlns.oracle.com/oracleas/schema/dbserver_audittrail-10_2.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://xmlns.oracle.com/oracleas/schema/dbserver_audittrail-10_2.xsd">
<Version>10.2</Version>
<AuditRecord>
    <Audit_Type>1</Audit_Type>
    <Session_Id>735807</Session_Id>
    <StatementId>9</StatementId>
    <EntryId>1</EntryId>
    <Extended_Timestamp>2010-05-02T23:14:49.399000</Extended_Timestamp>
    <DB_User>FRAMEWK</DB_User>
    <Client_Id>framewk</Client_Id>
    <OS_User>jsoap</OS_User>
    <Userhost>BIOA171</Userhost>
    <OS_Process>1680:6008</OS_Process>
    <Terminal>unknown</Terminal>
    <Instance_Number>0</Instance_Number>
    <Object_Schema>FRAMEWK</Object_Schema>
    <Object_Name>APP_USER</Object_Name>
    <Action>3</Action>
    <Returncode>0</Returncode>
    <Scn>1098595781</Scn>
    <Sql_Text>SELECT ... FROM ...</Sql_Text>
</AuditRecord>
</Audit>

[bfaber]

Lowell's answer here looks pretty sane. Remember that once the breaking is done, you can pipe results to xmlkv for some pretty nifty automatic field extraction.

[Lowell · Accepted solution]

I have a number of oracle related sourcetypes defined in a custom app, but I haven't come across this one before. Here is an XML config that I've modified to match your given log file sample.

Entries for props.conf:

[source::...[/\\][Oo]racle...[/\\]logs?...[/\\]audit.xml]
sourcetype = oracle_audit_xml

[oracle_audit_xml]
TIME_PREFIX = <Extended_Timestamp>
MAX_TIMESTAMP_LOOKAHEAD = 200
BREAK_ONLY_BEFORE = ^<AuditRecord>
BREAK_ONLY_BEFORE_DATE = False
SHOULD_LINEMERGE = True

Notes:

• You will need to update the [source::] pattern based on the file names of these log files.
• I'm assuming that if you have multiple log entries in a single file, then it's the <AuditRecord> will occur multiple times. Event breaking with XML is klugy, since XML was never meant for logs (IMHO), and it can't properly exist in fragments, so you'll probably end up with some extra junk before/after some of the very first or very last log entries.

If you provide a multiple-event example, someone could probably give you a better starting point.

[Lowell]

Whoops, that should be "rex" not "regex" in the example command above.

[Lowell]

You can do this at search time by doing something like this: sourcetype=oracle_audit_xml | xmlkv | regex field=Sql_Text mode=sed "s/[\r\n]+/ /g" Basically we are just replacing any end-of-line characters with a single space. Another approach would be to modify the raw event at index time, but that would be more difficult and I wouldn't recommend it unless you have some really good reasons.

[cparham]

Any suggestions on how to remove line breaks found inside elements .... Many of the SQL statements inside tags have hard line breaks often at very undesirable places.

[John_Mark]

I think you have to define it as a source type - and use the example above as a guide.