Getting Data In

XML log source type

cparham
Explorer

I cannot find much helpful documentation on handling XML log files. This link seems to be on the right track but what happens if I have multiple log file formats with different element names: http://www.splunk.com/wiki/Deploy:HowToWorkWithXMLLogFiles

Am I supposed to see xml_file as a source file type? I don't see it in the list of source types.

Here's what I'm wanting to parse. Its an Oracle database xml audit log:

<?xml version="1.0" encoding="UTF-8"?>
<Audit xmlns="http://xmlns.oracle.com/oracleas/schema/dbserver_audittrail-10_2.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://xmlns.oracle.com/oracleas/schema/dbserver_audittrail-10_2.xsd">
<Version>10.2</Version>
<AuditRecord>
    <Audit_Type>1</Audit_Type>
    <Session_Id>735807</Session_Id>
    <StatementId>9</StatementId>
    <EntryId>1</EntryId>
    <Extended_Timestamp>2010-05-02T23:14:49.399000</Extended_Timestamp>
    <DB_User>FRAMEWK</DB_User>
    <Client_Id>framewk</Client_Id>
    <OS_User>jsoap</OS_User>
    <Userhost>BIOA171</Userhost>
    <OS_Process>1680:6008</OS_Process>
    <Terminal>unknown</Terminal>
    <Instance_Number>0</Instance_Number>
    <Object_Schema>FRAMEWK</Object_Schema>
    <Object_Name>APP_USER</Object_Name>
    <Action>3</Action>
    <Returncode>0</Returncode>
    <Scn>1098595781</Scn>
    <Sql_Text>SELECT ... FROM ...</Sql_Text>
</AuditRecord>
</Audit>
Tags (1)
1 Solution

Lowell
Super Champion

I have a number of oracle related sourcetypes defined in a custom app, but I haven't come across this one before. Here is an XML config that I've modified to match your given log file sample.

Entries for props.conf:

[source::...[/\\][Oo]racle...[/\\]logs?...[/\\]audit.xml]
sourcetype = oracle_audit_xml

[oracle_audit_xml]
TIME_PREFIX = <Extended_Timestamp>
MAX_TIMESTAMP_LOOKAHEAD = 200
BREAK_ONLY_BEFORE = ^<AuditRecord>
BREAK_ONLY_BEFORE_DATE = False
SHOULD_LINEMERGE = True

Notes:

  • You will need to update the [source::] pattern based on the file names of these log files.
  • I'm assuming that if you have multiple log entries in a single file, then it's the <AuditRecord> will occur multiple times. Event breaking with XML is klugy, since XML was never meant for logs (IMHO), and it can't properly exist in fragments, so you'll probably end up with some extra junk before/after some of the very first or very last log entries.

If you provide a multiple-event example, someone could probably give you a better starting point.

View solution in original post

bfaber
Communicator

Lowell's answer here looks pretty sane. Remember that once the breaking is done, you can pipe results to xmlkv for some pretty nifty automatic field extraction.

Lowell
Super Champion

I have a number of oracle related sourcetypes defined in a custom app, but I haven't come across this one before. Here is an XML config that I've modified to match your given log file sample.

Entries for props.conf:

[source::...[/\\][Oo]racle...[/\\]logs?...[/\\]audit.xml]
sourcetype = oracle_audit_xml

[oracle_audit_xml]
TIME_PREFIX = <Extended_Timestamp>
MAX_TIMESTAMP_LOOKAHEAD = 200
BREAK_ONLY_BEFORE = ^<AuditRecord>
BREAK_ONLY_BEFORE_DATE = False
SHOULD_LINEMERGE = True

Notes:

  • You will need to update the [source::] pattern based on the file names of these log files.
  • I'm assuming that if you have multiple log entries in a single file, then it's the <AuditRecord> will occur multiple times. Event breaking with XML is klugy, since XML was never meant for logs (IMHO), and it can't properly exist in fragments, so you'll probably end up with some extra junk before/after some of the very first or very last log entries.

If you provide a multiple-event example, someone could probably give you a better starting point.

View solution in original post

Lowell
Super Champion

Whoops, that should be "rex" not "regex" in the example command above.

0 Karma

Lowell
Super Champion

You can do this at search time by doing something like this: sourcetype=oracle_audit_xml | xmlkv | regex field=Sql_Text mode=sed "s/[\r\n]+/ /g" Basically we are just replacing any end-of-line characters with a single space. Another approach would be to modify the raw event at index time, but that would be more difficult and I wouldn't recommend it unless you have some really good reasons.

0 Karma

cparham
Explorer

Any suggestions on how to remove line breaks found inside elements .... Many of the SQL statements inside tags have hard line breaks often at very undesirable places.

0 Karma

John_Mark
Splunk Employee
Splunk Employee

I think you have to define it as a source type - and use the example above as a guide.

0 Karma