Getting Data In

IIS and Exchange Log Header Extraction

Path Finder

I have been running the latest Splunk 4.1.1 and have been unsuccessful at getting the auto header extraction to work for IIS or Exchange logs. I am collecting the logs via a Splunk light forwarding agent installed on each of the servers, and then collecting the logs on my main Splunk server (Windows 2008 64bit). From all of the forums and docs, it looks like the file I should be editing is the props.conf located in Splunk\etc\system\local (on the main Splunk indexing server). I have added the following entries:

[ExchangeMessageTracking]
pulldown_type = true 
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
CHECK_FOR_HEADER = True

[iis]
pulldown_type = true 
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
CHECK_FOR_HEADER = True

After making the changes I have tried restarting the Splunk services and running "| extract reload=true". I also tried placing these settings in Splunk\etc\apps\search\local. However, nothing seems to work. Splunk indexes the entire file, including the header lines that begin with a "#". If anyone has any suggestions I would really appreciate them.

Tags (3)
1 Solution

Contributor

I have created my manual extraction of the fields due to the fact that I've never had the CHECK_FOR_HEADER attribute to work. Here's my IIS config:

props.conf

[iis_default]
pulldown_type = true
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
REPORT-iis_default = iis_default

[iis_w3c]
pulldown_type = true
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
REPORT-iis_w3c = iis_w3c
TRANSFORMS-comment = comment

transforms.conf

[no_header]
REGEX = NetBIOSName,DNSName,IP,MAC,OS,AuditID,CVE,Name,Description,Date,Risk,CVSSScore,FixInformation
DEST_KEY = queue
FORMAT = nullQueue

[iis_default]
FIELDS="c-ip","cs-username","date","time","service","s-name","s-ip","time-taken","c-sent","s-sent","sc-status","sc_win_status","cs_method","cs-uri-stem"
DELIMS = ","

[iis_w3c]
FIELDS="date","time","c-ip","cs-username","s-sitename","s-ip","s-port","cs-method","cs-uri-stem","cs-uri-query","sc-status","sc-win32-status","cs(User-Agent)"
DELIMS = " "

View solution in original post

Splunk Employee
Splunk Employee

In Splunk 6, you can also use INDEXED_EXTRACTIONS=W3C which auto-extracts the fields from the headers at index-time.

http://docs.splunk.com/Documentation/Splunk/latest/Data/Extractfieldsfromfileheadersatindextime

0 Karma

Splunk Employee
Splunk Employee

I found a new format iis_v6, that seems to be for IIS version 6.0
here are my settings.

inputs
[montiror:]
sourcetype=iis_v6
TZ=GMT

in props.conf
[iis_v6]
pulldown_type = true
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
REPORT-iis_v6 = iis_v6
TRANSFORMS-comment = iis_comment


in transforms.conf
[iis_v6]
FIELDS="date","time","s-sitename","s-computername","s-ip","cs-method","cs-uri-stem","cs-uri-query","s-port","cs-username","c-ip","cs-version","cs_User-Agent_","cs_Cookie_","cs_Referer_","cs-host","sc-status","sc-substatus","sc-win32-status","sc-bytes","cs-bytes","time-taken"
DELIMS = " "

[iis_comment]
REGEX = ^#
DEST_KEY = queue
FORMAT = nullQueue

Splunk Employee
Splunk Employee

I have just uploaded Splunk App for Microsoft Exchange, which extracts the information you need. I hope it helps you.

0 Karma

Contributor

I have created my manual extraction of the fields due to the fact that I've never had the CHECK_FOR_HEADER attribute to work. Here's my IIS config:

props.conf

[iis_default]
pulldown_type = true
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
REPORT-iis_default = iis_default

[iis_w3c]
pulldown_type = true
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
REPORT-iis_w3c = iis_w3c
TRANSFORMS-comment = comment

transforms.conf

[no_header]
REGEX = NetBIOSName,DNSName,IP,MAC,OS,AuditID,CVE,Name,Description,Date,Risk,CVSSScore,FixInformation
DEST_KEY = queue
FORMAT = nullQueue

[iis_default]
FIELDS="c-ip","cs-username","date","time","service","s-name","s-ip","time-taken","c-sent","s-sent","sc-status","sc_win_status","cs_method","cs-uri-stem"
DELIMS = ","

[iis_w3c]
FIELDS="date","time","c-ip","cs-username","s-sitename","s-ip","s-port","cs-method","cs-uri-stem","cs-uri-query","sc-status","sc-win32-status","cs(User-Agent)"
DELIMS = " "

View solution in original post

Super Champion

It is an IIS log. IIS. Windows IIS. This is about as common as logs are. How about an out of the box solution? Just sayin.

0 Karma

Splunk Employee
Splunk Employee
0 Karma

Path Finder

BunnyHop, I got the extraction to work by using your suggestion and I also got some additional information from this forum post:
http://www.splunk.com/support/forum:SplunkAdministration/3395

For anyone else with this problem, I only made changes on the main Splunk server and not on a forwarder. I also did not need to restart the Splunk service(s) in order to see the changes take effect.

0 Karma