Webknight Field Extractions and Header Exclusions

[hughroberts]

If anybody uses WebKnight ISAPA filter in your environment you will probably have spotted that the log file formal can take a bit of cajoling to import neatly.

As I spent a few long hours getting the the following configuration right to make the field extraction work neatly, I wanted to share it with the community to save others some time !

The main challenges I encountered with the file format are:

a) Multiple quote lines at the start of the each log file.

b) Header line in a quote line with a superfluous field tag.

c) Writes to multiple log file names that have the date and other variables in the file name (if you config WebKnight to do this).

d) Splunk imports the quote lines as one multiple event.

e) The date and time information is in two separate fields that can confuse spunk into thinking that each field couple is a field name and field combination.

[hughroberts]

Here are the inputs, props and transforms for your set up. The inputs.conf goes onto the wherever your UniversalForwarder is installed. The others go on to the indexer/search, you need to put the stanzas to eliminate headers in place before you index the data, the field extractions are only applied at search time.

Tested on versions 5.0.3 and 5.0.5

Happy Splunking !

<< inputs.conf >>

[default]
host = WEBSERVER

[monitor://C:\webknight/App.*]
sourcetype=webknight
index=webknight-index
disabled=0

<< props.conf >>

[source::C:\webknight/App.*]
sourcetype=webknight

[webknight]
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE=false
REPORT-webknightextract = webknight_extractions
TRANSFORMS-t1=eliminate_header

<< transforms.conf >>

[webknight_extractions]
DELIMS=";"
FIELDS=WAFDate,WAFTime,WAFInst,WAFEvent,WAFIPA,WAFUser,WAFHost,WAFAgent,WAFAdditions1

[eliminate_header]
REGEX=^(?:#Software:|#Date:|#LogTime:|#Fields:)\s
DEST_KEY=queue
FORMAT=nullQueue