Ignoring Comments in Log Files

[enielson]

I am trying to index the httperr files from IIS, located in C:\windows\system32\LogFiles\HTTPERR but each of those files start with a set of lines that begin with a #.

How can I instruct splunk to ignore lines starting with #?

[paul_1994]

Was this done on the indexer? or the search head? It seems like it should happen on the indexer but I wanted to make sure..

[paul_1994]

Thanks. I made the change on my indexers and it seems to be working in our test environment.

[Paolo_Prigione]

The filtering has to be configured by the instance which parses the data. If you have intermediate heavy forwarder, they are parsing. If the Universal forwarder sends directly to indexers, they are the ones parsing.

[silvermail · Accepted solution]

Adding to the answer, this is what I am using.

props.conf

[iis]
KV_MODE = none
CHECK_FOR_HEADER = false
TRANSFORMS-commentsToNull = iis-ignore-comments
REPORT-iisfields = iis-w3c-extended-fields
TIME_FORMAT = %Y-%m-%d %H:%M:%S

transforms.conf

[iis-ignore-comments]
REGEX = ^#(?:Version|Software|Fields|Date):\s.*$
DEST_KEY = queue
FORMAT = nullQueue

[slierninja]

This worked out beautifully, I also had to add iis-2, iis-10, and other iis-* source types that I was using to props.conf.

[dwaddle]

You could route the undesired event(s) to the nullQueue - see these for more info:

http://answers.splunk.com/questions/96/how-do-i-exclude-some-events-from-being-indexed-by-splunk

http://www.splunk.com/base/Documentation/4.1.5/Admin/Routeandfilterdata