I have been running the latest Splunk 4.1.1 and have been unsuccessful at getting the auto header extraction to work for IIS or Exchange logs. I am collecting the logs via a Splunk light forwarding agent installed on each of the servers, and then collecting the logs on my main Splunk server (Windows 2008 64bit). From all of the forums and docs, it looks like the file I should be editing is the props.conf located in Splunk\etc\system\local (on the main Splunk indexing server). I have added the following entries:
[ExchangeMessageTracking]
pulldown_type = true
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
CHECK_FOR_HEADER = True
[iis]
pulldown_type = true
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
CHECK_FOR_HEADER = True
After making the changes I have tried restarting the Splunk services and running "| extract reload=true". I also tried placing these settings in Splunk\etc\apps\search\local. However, nothing seems to work. Splunk indexes the entire file, including the header lines that begin with a "#". If anyone has any suggestions I would really appreciate them.
I have created my manual extraction of the fields due to the fact that I've never had the CHECK_FOR_HEADER attribute to work. Here's my IIS config:
props.conf
[iis_default] pulldown_type = true MAX_TIMESTAMP_LOOKAHEAD = 32 SHOULD_LINEMERGE = False REPORT-iis_default = iis_default [iis_w3c] pulldown_type = true MAX_TIMESTAMP_LOOKAHEAD = 32 SHOULD_LINEMERGE = False REPORT-iis_w3c = iis_w3c TRANSFORMS-comment = comment
transforms.conf
[no_header] REGEX = NetBIOSName,DNSName,IP,MAC,OS,AuditID,CVE,Name,Description,Date,Risk,CVSSScore,FixInformation DEST_KEY = queue FORMAT = nullQueue [iis_default] FIELDS="c-ip","cs-username","date","time","service","s-name","s-ip","time-taken","c-sent","s-sent","sc-status","sc_win_status","cs_method","cs-uri-stem" DELIMS = "," [iis_w3c] FIELDS="date","time","c-ip","cs-username","s-sitename","s-ip","s-port","cs-method","cs-uri-stem","cs-uri-query","sc-status","sc-win32-status","cs(User-Agent)" DELIMS = " "
In Splunk 6, you can also use INDEXED_EXTRACTIONS=W3C which auto-extracts the fields from the headers at index-time.
http://docs.splunk.com/Documentation/Splunk/latest/Data/Extractfieldsfromfileheadersatindextime
I found a new format iis_v6, that seems to be for IIS version 6.0
here are my settings.
inputs [montiror:] sourcetype=iis_v6 TZ=GMT in props.conf [iis_v6] pulldown_type = true MAX_TIMESTAMP_LOOKAHEAD = 32 SHOULD_LINEMERGE = False REPORT-iis_v6 = iis_v6 TRANSFORMS-comment = iis_comment in transforms.conf [iis_v6] FIELDS="date","time","s-sitename","s-computername","s-ip","cs-method","cs-uri-stem","cs-uri-query","s-port","cs-username","c-ip","cs-version","cs_User-Agent_","cs_Cookie_","cs_Referer_","cs-host","sc-status","sc-substatus","sc-win32-status","sc-bytes","cs-bytes","time-taken" DELIMS = " " [iis_comment] REGEX = ^# DEST_KEY = queue FORMAT = nullQueue
I have just uploaded Splunk App for Microsoft Exchange, which extracts the information you need. I hope it helps you.
I have created my manual extraction of the fields due to the fact that I've never had the CHECK_FOR_HEADER attribute to work. Here's my IIS config:
props.conf
[iis_default] pulldown_type = true MAX_TIMESTAMP_LOOKAHEAD = 32 SHOULD_LINEMERGE = False REPORT-iis_default = iis_default [iis_w3c] pulldown_type = true MAX_TIMESTAMP_LOOKAHEAD = 32 SHOULD_LINEMERGE = False REPORT-iis_w3c = iis_w3c TRANSFORMS-comment = comment
transforms.conf
[no_header] REGEX = NetBIOSName,DNSName,IP,MAC,OS,AuditID,CVE,Name,Description,Date,Risk,CVSSScore,FixInformation DEST_KEY = queue FORMAT = nullQueue [iis_default] FIELDS="c-ip","cs-username","date","time","service","s-name","s-ip","time-taken","c-sent","s-sent","sc-status","sc_win_status","cs_method","cs-uri-stem" DELIMS = "," [iis_w3c] FIELDS="date","time","c-ip","cs-username","s-sitename","s-ip","s-port","cs-method","cs-uri-stem","cs-uri-query","sc-status","sc-win32-status","cs(User-Agent)" DELIMS = " "
It is an IIS log. IIS. Windows IIS. This is about as common as logs are. How about an out of the box solution? Just sayin.
FYI CHECK_FOR_HEADER is deprecated since 5.0.*
see http://docs.splunk.com/Documentation/Splunk/5.0.4/ReleaseNotes/DeprecatedFeatures
BunnyHop, I got the extraction to work by using your suggestion and I also got some additional information from this forum post:
http://www.splunk.com/support/forum:SplunkAdministration/3395
For anyone else with this problem, I only made changes on the main Splunk server and not on a forwarder. I also did not need to restart the Splunk service(s) in order to see the changes take effect.