topic Re: Ignoring Comments in Log Files in Getting Data In

Ignoring Comments in Log Files

enielson — Fri, 26 Nov 2010 04:20:12 GMT

I am trying to index the httperr files from IIS, located in C:\windows\system32\LogFiles\HTTPERR but each of those files start with a set of lines that begin with a #.

How can I instruct splunk to ignore lines starting with #?

Re: Ignoring Comments in Log Files

dwaddle — Fri, 26 Nov 2010 05:59:25 GMT

You could route the undesired event(s) to the nullQueue - see these for more info:

http://answers.splunk.com/questions/96/how-do-i-exclude-some-events-from-being-indexed-by-splunk

http://www.splunk.com/base/Documentation/4.1.5/Admin/Routeandfilterdata

Re: Ignoring Comments in Log Files

silvermail — Fri, 26 Nov 2010 23:16:29 GMT

Adding to the answer, this is what I am using.

props.conf

[iis]
KV_MODE = none
CHECK_FOR_HEADER = false
TRANSFORMS-commentsToNull = iis-ignore-comments
REPORT-iisfields = iis-w3c-extended-fields
TIME_FORMAT = %Y-%m-%d %H:%M:%S

transforms.conf

[iis-ignore-comments]
REGEX = ^#(?:Version|Software|Fields|Date):\s.*$
DEST_KEY = queue
FORMAT = nullQueue

Re: Ignoring Comments in Log Files

paul_1994 — Sun, 23 Sep 2012 18:36:49 GMT

Was this done on the indexer? or the search head? It seems like it should happen on the indexer but I wanted to make sure..

Re: Ignoring Comments in Log Files

Paolo_Prigione — Mon, 24 Sep 2012 09:43:07 GMT

The filtering has to be configured by the instance which parses the data. If you have intermediate heavy forwarder, they are parsing. If the Universal forwarder sends directly to indexers, they are the ones parsing.

Re: Ignoring Comments in Log Files

paul_1994 — Mon, 24 Sep 2012 16:55:16 GMT

Thanks. I made the change on my indexers and it seems to be working in our test environment.

Re: Ignoring Comments in Log Files

slierninja — Thu, 17 Jan 2013 14:50:38 GMT

This worked out beautifully, I also had to add iis-2, iis-10, and other iis-* source types that I was using to props.conf.