Splunk Search
Highlighted

DELIM does not work

Path Finder

We can not get field extraction to work with IIS log files. Any suggestions?

transforms.conf
[iisw3cfields]
DELIMS = " "
FIELDS = date,time,s-computername,s-ip,cs-method,cs-uri-stem,cs-uri-query,s-port,cs-username,c-ip,cs-version,cs(User-Agent),cs(Cookie),cs(Referer),cs-host,sc-status,sc-bytes,cs-bytes,time-taken

props.conf (relevant part)
...
[iisex]
TIME
PREFIX = ^
MAXTIMESTAMPLOOKAHEAD = 19
TIMEFORMAT = %Y-%m-%d %H:%M:%S
TZ = GMT
LINE
BREAKER = ([\r\n]+)(\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2})
SHOULDLINEMERGE = False
TRUNCATE = 10000
KV
MODE = None
pulldown_type = true
REPORT-iisw3cfields = iisw3cfields

Here are a few lines from our log file ...

Software: Microsoft Internet Information Services 6.0

Version: 1.0

Date: 2013-07-07 00:00:00

Fields: date time s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Cookie) cs(Referer) cs-host sc-status sc-bytes cs-bytes time-taken

2013-07-06 23:59:59 TLOWPN29 192.168.122.82 GET /secure/EmployeeHome.aspx - 443 - 207.190.231.139 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+5.2;+WOW64;+.NET+CLR+2.0.50727;+.NET+CLR+3.0.4506.2152;+.NET+CLR+3.5.30729) training=true;+SessionGUID=ee81cbbf-9da8-4532-92ae-24a4e8ad6147;+StyleSheetTheme=pxxxx;+SSODeferral=true hxxx://xxxxx.xxxx.com/secure/login.aspx?alias=0414T735 xxxxxxx.xxxxx.com 200 68499 649 1109

2013-07-07 00:00:00 TLOWPN29 192.168.122.82 GET /secure/Images/exception.gif - 443 - 38.102.232.83 Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+5.1;+Trident/4.0;+.NET+CLR+1.1.4322;+.NET+CLR+2.0.50727;+.NET+CLR+3.0.4506.2152;+.NET+CLR+3.5.30729;+.NET4.0C;+.NET4.0E) StyleSheetTheme=pxxxx;+SessionGUID=00000000-0000-0000-0000-000000000000;+training=true hxxps://xxxxxxx.xxxx.com/secure/login.aspx tixxxxxx.xxxxx.com 200 1338 597 15

0 Karma
Highlighted

Re: DELIM does not work

Champion

i think it's overcooked. Let it flow into splunk. Splunk should be able to get the events divided whereever it finds time at the start. After that, do the field extraction on the UI, which is easier, where you can actually see the matching. If the time is not correct, set the Props.conf for time recognition. Thanks

0 Karma
Highlighted

Re: DELIM does not work

Champion

I seen this problem before. The problem is not with your delim, but with your field names in FIELDS. Change your "-" to "_" or remove "-" entirely. Splunk tends to not like the dash when naming fields.

Example changes:

  • cs-host to cs_host
  • cs(User-Agent) to cs_userAgent . Personally I typically use all lower case.
  • cs(Referer) to cs_referer

Doing this should fix your problem.

Also this simplified props file that works.


[iis_ex]
pulldown_type=true
MAX_TIMESTAMP_LOOKAHEAD=19
TIME_FORMAT=%Y-%m-%d %H:%M:%S
SHOULD_LINEMERGE=false
CHECK_FOR_HEADER=false
TZ=GMT
REPORT-iisw3cfields=iisw3cfields

Hope this help or gets you started cheers.

0 Karma
Highlighted

Re: DELIM does not work

Splunk Employee
Splunk Employee

If you are using Splunk 6 on both Forwarder and Indexer (or just ingesting logs locally on the indexer) set sourcetype=iis in your inputs.conf.

Under the covers, this is using INDEXED_EXTRACTIONS=W3C in props.conf and will automatically pick up the header and use it for field mappings so you don't have to mess with props and transforms.

http://docs.splunk.com/Documentation/Splunk/latest/Data/Extractfieldsfromfileheadersatindextime

0 Karma