topic Re: IIS and Exchange Log Header Extraction in Getting Data In

IIS and Exchange Log Header Extraction

Justin — Wed, 28 Apr 2010 05:08:04 GMT

I have been running the latest Splunk 4.1.1 and have been unsuccessful at getting the auto header extraction to work for IIS or Exchange logs. I am collecting the logs via a Splunk light forwarding agent installed on each of the servers, and then collecting the logs on my main Splunk server (Windows 2008 64bit). From all of the forums and docs, it looks like the file I should be editing is the props.conf located in Splunk\etc\system\local (on the main Splunk indexing server). I have added the following entries:

[ExchangeMessageTracking]
pulldown_type = true 
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
CHECK_FOR_HEADER = True

[iis]
pulldown_type = true 
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
CHECK_FOR_HEADER = True

After making the changes I have tried restarting the Splunk services and running "| extract reload=true". I also tried placing these settings in Splunk\etc\apps\search\local. However, nothing seems to work. Splunk indexes the entire file, including the header lines that begin with a "#". If anyone has any suggestions I would really appreciate them.

Re: IIS and Exchange Log Header Extraction

BunnyHop — Wed, 28 Apr 2010 19:53:15 GMT

I have created my manual extraction of the fields due to the fact that I've never had the CHECK_FOR_HEADER attribute to work. Here's my IIS config:

props.conf

[iis_default]
pulldown_type = true
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
REPORT-iis_default = iis_default

[iis_w3c]
pulldown_type = true
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
REPORT-iis_w3c = iis_w3c
TRANSFORMS-comment = comment

transforms.conf

[no_header]
REGEX = NetBIOSName,DNSName,IP,MAC,OS,AuditID,CVE,Name,Description,Date,Risk,CVSSScore,FixInformation
DEST_KEY = queue
FORMAT = nullQueue

[iis_default]
FIELDS="c-ip","cs-username","date","time","service","s-name","s-ip","time-taken","c-sent","s-sent","sc-status","sc_win_status","cs_method","cs-uri-stem"
DELIMS = ","

[iis_w3c]
FIELDS="date","time","c-ip","cs-username","s-sitename","s-ip","s-port","cs-method","cs-uri-stem","cs-uri-query","sc-status","sc-win32-status","cs(User-Agent)"
DELIMS = " "

Re: IIS and Exchange Log Header Extraction

Justin — Thu, 29 Apr 2010 01:47:07 GMT

BunnyHop, I got the extraction to work by using your suggestion and I also got some additional information from this forum post:
http://www.splunk.com/support/forum:SplunkAdministration/3395

For anyone else with this problem, I only made changes on the main Splunk server and not on a forwarder. I also did not need to restart the Splunk service(s) in order to see the changes take effect.

Re: IIS and Exchange Log Header Extraction

ahall_splunk — Mon, 15 Aug 2011 18:09:08 GMT

I have just uploaded Splunk App for Microsoft Exchange, which extracts the information you need. I hope it helps you.

Re: IIS and Exchange Log Header Extraction

yannK — Fri, 14 Oct 2011 19:11:40 GMT

I found a new format iis_v6, that seems to be for IIS version 6.0
here are my settings.

inputs
[montiror:]
sourcetype=iis_v6
TZ=GMT

in props.conf
[iis_v6]
pulldown_type = true
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
REPORT-iis_v6 = iis_v6
TRANSFORMS-comment = iis_comment


in transforms.conf
[iis_v6]
FIELDS="date","time","s-sitename","s-computername","s-ip","cs-method","cs-uri-stem","cs-uri-query","s-port","cs-username","c-ip","cs-version","cs_User-Agent_","cs_Cookie_","cs_Referer_","cs-host","sc-status","sc-substatus","sc-win32-status","sc-bytes","cs-bytes","time-taken"
DELIMS = " "

[iis_comment]
REGEX = ^#
DEST_KEY = queue
FORMAT = nullQueue

Re: IIS and Exchange Log Header Extraction

yannK — Mon, 28 Sep 2020 14:40:02 GMT

FYI CHECK_FOR_HEADER is deprecated since 5.0.*
see http://docs.splunk.com/Documentation/Splunk/5.0.4/ReleaseNotes/DeprecatedFeatures

Re: IIS and Exchange Log Header Extraction

lukejadamec — Wed, 28 Aug 2013 00:10:29 GMT

It is an IIS log. IIS. Windows IIS. This is about as common as logs are. How about an out of the box solution? Just sayin.

Re: IIS and Exchange Log Header Extraction

ogdin — Thu, 13 Feb 2014 16:33:44 GMT

In Splunk 6, you can also use INDEXED_EXTRACTIONS=W3C which auto-extracts the fields from the headers at index-time.

http://docs.splunk.com/Documentation/Splunk/latest/Data/Extractfieldsfromfileheadersatindextime