topic Re: IIS log fields not parsing in Getting Data In

IIS log fields not parsing

twinspop — Tue, 24 May 2011 17:18:06 GMT

Windows 2003 with SUF, inputs.conf:

[monitor://C:\WINNT\system32\LogFiles\HTTPERR]
disabled = false
sourcetype = iis

Linux running the indexer, props.conf:

[iis]
CHECK_FOR_HEADER = true
[iis-2]
CHECK_FOR_HEADER = true

Based on testing with oneshot uploads beforehand, I didn't need either of those but added them during troubleshooting. The logs are being tagged correctly by the forwarder with an iis sourcetype -- iis-2 in this case. The fields are not being identified however.

If I upload one of the log files from the forwarder into the indexer via the web interface and specify iis as the sourcetype, the fields are extracted as I'd expect. (The uploaded file gets tagged as iis-3 FWIW.)

Why are the Universal Forwarder logs being handled differently and how to fix it?

Thanks,
jon

Re: IIS log fields not parsing

ftk — Tue, 24 May 2011 18:54:58 GMT

I have run into the same problem, and ended up specifying the extractions manually. The following will work for IIS W3C (with all fields enabled) as well as the HTTPERR log (which is what you are looking to index):

props.conf:

[iisw3c]
pulldown_type = true 
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
CHECK_FOR_HEADER = False
TZ = GMT
REPORT-iisw3cfields = iisw3cfields
TRANSFORMS-removecomments = removecomments

[iishttperr]
pulldown_type = true 
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
CHECK_FOR_HEADER = False
TZ = GMT
REPORT-iishttperrfields = iishttperrfields
TRANSFORMS-removecomments = removecomments

transforms.conf

[removecomments]
REGEX = ^\#.*
DEST_KEY = queue
FORMAT = nullQueue

# IIS W3C Log field extractions (Identical in IIS 6 and 7)
# These assume that you have enabled all available fields to be logged
[iisw3cfields]
DELIMS = " "
FIELDS = date,time,s-sitename,s-computername,s-ip,cs-method,cs-uri-stem,cs-uri-query,s-port,cs-username,c-ip,cs-version,cs(User-Agent),cs(Cookie),cs(Referer),cs-host,sc-status,sc-substatus,sc-win32-status,sc-bytes,cs-bytes,time-taken

[iishttperrfields]
DELIMS = " "
FIELDS = date,time,c-ip,c-port,s-ip,s-port,cs-version,cs-method,cs-uri,sc-status,s-siteid,s-reason,s-queuename

and then specify a sourcetype of [iisw3c] or [iishttperr] for your monitor in inputs.conf.

Re: IIS log fields not parsing

twinspop — Tue, 24 May 2011 20:04:30 GMT

Well it's nice to see I'm not the only one on this one. 🙂 Thanks for the workarounds. Has Splunk Support weighed in on this? Seems like a bug to me, unless you and I both are missing something.

Re: IIS log fields not parsing

ftk — Tue, 24 May 2011 20:56:28 GMT

I actually don't remember if I ever filed anything for this or not, but this seems to be a fairly common occurrence.

Re: IIS log fields not parsing

markrobinson734 — Wed, 10 Aug 2011 18:44:46 GMT

Had the same problem. This worked for me, though I had to change the required field as we weren't monitoring all. Thanks!

Re: IIS log fields not parsing

Ayn — Mon, 26 Sep 2011 09:08:46 GMT

I'm having the same issue. Is this filed as a bug anywhere?

Re: IIS log fields not parsing

hexx — Mon, 28 Sep 2020 09:56:29 GMT

This is a known limitation of CHECK_FOR_HEADER in a distributed environment.

From props.conf.spec :

CHECK_FOR_HEADER = [true|false]
* Used for index-time field extractions only.

(...)

* The field names are stored in etc/apps/learned/local/props.conf.
  * Because of this, this feature will not work in most environments where the data is forwarded.

In your case, CHECK_FOR_HEADER writes the header-based field extractions in the learned app of the forwarder, information which is of course not available when you search from the indexer.

Of the two work-arounds discussed here, my preference goes to ftk's (the second one listed) :

1 - Ingest a couple of files on the indexer to create the field extractions there for sourcetypes iis and iis-2 at the minimum.

2 - Disable CHECK_FOR_HEADER on the forwarder for this sourcetype, clean up the learned app and configure the delimiter-based field extraction manually in transforms.conf on the indexer or search-head.