IIS and Exchange Log Header Extraction

[Justin]

I have been running the latest Splunk 4.1.1 and have been unsuccessful at getting the auto header extraction to work for IIS or Exchange logs. I am collecting the logs via a Splunk light forwarding agent installed on each of the servers, and then collecting the logs on my main Splunk server (Windows 2008 64bit). From all of the forums and docs, it looks like the file I should be editing is the props.conf located in Splunk\etc\system\local (on the main Splunk indexing server). I have added the following entries:

[ExchangeMessageTracking]
pulldown_type = true 
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
CHECK_FOR_HEADER = True

[iis]
pulldown_type = true 
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
CHECK_FOR_HEADER = True

After making the changes I have tried restarting the Splunk services and running "| extract reload=true". I also tried placing these settings in Splunk\etc\apps\search\local. However, nothing seems to work. Splunk indexes the entire file, including the header lines that begin with a "#". If anyone has any suggestions I would really appreciate them.

[ogdin]

In Splunk 6, you can also use INDEXED_EXTRACTIONS=W3C which auto-extracts the fields from the headers at index-time.

http://docs.splunk.com/Documentation/Splunk/latest/Data/Extractfieldsfromfileheadersatindextime

[yannK]

I found a new format iis_v6, that seems to be for IIS version 6.0
here are my settings.

inputs
[montiror:]
sourcetype=iis_v6
TZ=GMT

in props.conf
[iis_v6]
pulldown_type = true
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
REPORT-iis_v6 = iis_v6
TRANSFORMS-comment = iis_comment


in transforms.conf
[iis_v6]
FIELDS="date","time","s-sitename","s-computername","s-ip","cs-method","cs-uri-stem","cs-uri-query","s-port","cs-username","c-ip","cs-version","cs_User-Agent_","cs_Cookie_","cs_Referer_","cs-host","sc-status","sc-substatus","sc-win32-status","sc-bytes","cs-bytes","time-taken"
DELIMS = " "

[iis_comment]
REGEX = ^#
DEST_KEY = queue
FORMAT = nullQueue

[ahall_splunk]

I have just uploaded Splunk App for Microsoft Exchange, which extracts the information you need. I hope it helps you.

[BunnyHop · Accepted solution]

I have created my manual extraction of the fields due to the fact that I've never had the CHECK_FOR_HEADER attribute to work. Here's my IIS config:

props.conf

[iis_default]
pulldown_type = true
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
REPORT-iis_default = iis_default

[iis_w3c]
pulldown_type = true
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
REPORT-iis_w3c = iis_w3c
TRANSFORMS-comment = comment

transforms.conf

[no_header]
REGEX = NetBIOSName,DNSName,IP,MAC,OS,AuditID,CVE,Name,Description,Date,Risk,CVSSScore,FixInformation
DEST_KEY = queue
FORMAT = nullQueue

[iis_default]
FIELDS="c-ip","cs-username","date","time","service","s-name","s-ip","time-taken","c-sent","s-sent","sc-status","sc_win_status","cs_method","cs-uri-stem"
DELIMS = ","

[iis_w3c]
FIELDS="date","time","c-ip","cs-username","s-sitename","s-ip","s-port","cs-method","cs-uri-stem","cs-uri-query","sc-status","sc-win32-status","cs(User-Agent)"
DELIMS = " "

[lukejadamec]

It is an IIS log. IIS. Windows IIS. This is about as common as logs are. How about an out of the box solution? Just sayin.

[yannK]

FYI CHECK_FOR_HEADER is deprecated since 5.0.*
see http://docs.splunk.com/Documentation/Splunk/5.0.4/ReleaseNotes/DeprecatedFeatures

[Justin]

BunnyHop, I got the extraction to work by using your suggestion and I also got some additional information from this forum post:
http://www.splunk.com/support/forum:SplunkAdministration/3395

For anyone else with this problem, I only made changes on the main Splunk server and not on a forwarder. I also did not need to restart the Splunk service(s) in order to see the changes take effect.