Record Event Break

[rjsteele]

It does not look like Splunk is breaking my XML correctly. I’d like to break it before each ‘<record version=’ tag in the XML. I am trying to set the props.conf with the following entry, but it does not seem to affect the way Splunk breaks the already indexed data, even after I stop and start Splunk. Can anyone assist?

<source::/auditserverDAT/audit*>
LINE_BREAKER=<record version

Here is a sample XML file with a few records:

<?xml version='1.0' encoding='UTF-8' ?>
<?xml-stylesheet type='text/xsl' href='file:///usr/share/lib/xml/style/adt_record.xsl.1' ?>
<!DOCTYPE audit PUBLIC '-//Sun Microsystems, Inc.//DTD Audit V1//EN' 'file:///usr/share/lib/xml/dtd/adt_record.dtd.1'>
<audit>
<file iso8601="2007-08-21 10:35:46.111 +08:00">/var/audit/20070821023545.20070821023546.chihung</file>
<record version="2" event="system booted" modifier="na" iso8601="2007-08-21 10:34:49.704 +08:00">
<text>booting kernel</text>
</record>
<record version="2" event="stat(2)" host="chihung" iso8601="2007-08-21 10:39:22.700 +08:00">
<path>/usr/lib/pt_chmod</path>
<attribute mode="104511" uid="root" gid="bin" fsid="85" nodeid="623" device="0"/>
<subject audit-uid="chihung" uid="root" gid="staff" ruid="chihung" rgid="staff" pid="704" sid="2477344358" tid="756 65558 ftpl_2_207"/>
<return errval="success" retval="0"/>
</record>
<record version="2" event="stat(2)" modifier="fe" host="chihung" iso8601="2007-08-21 10:39:22.727 +08:00">
<path>/platform/SUNW,UltraSPARC-IIi-cEngine/lib</path>
<subject audit-uid="chihung" uid="root" gid="staff" ruid="chihung" rgid="staff" pid="704" sid="2477344358" tid="756 65558 ftpl_2_207"/>
<return errval="failure: No such file or directory

[acharlieh · Accepted solution]

Breaking a stream of data into lines, and aggregating that data back into events happens only once, at index time. If you have events already indexed by Splunk, these will not be affected by changes to line breaking or aggregation settings. You could hide or remove existing data and reindex them to apply new settings.

Now regarding your props.conf, if you're wanting to use LINE_BREAKER to split events, you must include a capturing group which is where the division of the line actually occurs, and you should also set SHOULD_LINEMERGE = false so that Splunk doesn't attempt to merge these "lines" back into larger events. You could also consider instead using BREAK_ONLY_BEFORE and other attributes to determine event boundaries.

An excellent reference on at what part of indexing and parsing each attribute is referenced can be found on the splunk wiki. The splunk docs contain a bunch of pages on event processing as well. There is also a guide on how to remove already indexed data from Splunk with options ranging from hiding individual events to destroying entire indexes.

[rjsteele]

This seemed to work.

splunk clean eventdata -index <index>

Entered the following in prop.cond

<source::/auditserverDAT/audit*>
BREAK_ONLY_BEFORE=<record version

[rjsteele]

Thanks acharlieh. I will look into the references. I do not want to split event, sorry for the confusion. I simply want Splunk to use '