Splunk Search

DELIM does not work

wsnyder2
Path Finder

We can not get field extraction to work with IIS log files. Any suggestions?

transforms.conf
[iisw3cfields]
DELIMS = " "
FIELDS = date,time,s-computername,s-ip,cs-method,cs-uri-stem,cs-uri-query,s-port,cs-username,c-ip,cs-version,cs(User-Agent),cs(Cookie),cs(Referer),cs-host,sc-status,sc-bytes,cs-bytes,time-taken

props.conf (relevant part)
...
[iis_ex]
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 19
TIME_FORMAT = %Y-%m-%d %H:%M:%S
TZ = GMT
LINE_BREAKER = ([\r\n]+)(\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2})
SHOULD_LINEMERGE = False
TRUNCATE = 10000
KV_MODE = None
pulldown_type = true
REPORT-iisw3cfields = iisw3cfields

Here are a few lines from our log file ...

Software: Microsoft Internet Information Services 6.0

Version: 1.0

Date: 2013-07-07 00:00:00

Fields: date time s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Cookie) cs(Referer) cs-host sc-status sc-bytes cs-bytes time-taken

2013-07-06 23:59:59 TLOWPN29 192.168.122.82 GET /secure/EmployeeHome.aspx - 443 - 207.190.231.139 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+5.2;+WOW64;+.NET+CLR+2.0.50727;+.NET+CLR+3.0.4506.2152;+.NET+CLR+3.5.30729) training=true;+SessionGUID=ee81cbbf-9da8-4532-92ae-24a4e8ad6147;+StyleSheetTheme=pxxxx;+SSODeferral=true hxxx://xxxxx.xxxx.com/secure/login.aspx?alias=0414T735 xxxxxxx.xxxxx.com 200 68499 649 1109

2013-07-07 00:00:00 TLOWPN29 192.168.122.82 GET /secure/Images/exception.gif - 443 - 38.102.232.83 Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+5.1;+Trident/4.0;+.NET+CLR+1.1.4322;+.NET+CLR+2.0.50727;+.NET+CLR+3.0.4506.2152;+.NET+CLR+3.5.30729;+.NET4.0C;+.NET4.0E) StyleSheetTheme=pxxxx;+SessionGUID=00000000-0000-0000-0000-000000000000;+training=true hxxps://xxxxxxx.xxxx.com/secure/login.aspx tixxxxxx.xxxxx.com 200 1338 597 15

0 Karma

ogdin
Splunk Employee
Splunk Employee

If you are using Splunk 6 on both Forwarder and Indexer (or just ingesting logs locally on the indexer) set sourcetype=iis in your inputs.conf.

Under the covers, this is using INDEXED_EXTRACTIONS=W3C in props.conf and will automatically pick up the header and use it for field mappings so you don't have to mess with props and transforms.

http://docs.splunk.com/Documentation/Splunk/latest/Data/Extractfieldsfromfileheadersatindextime

0 Karma

bmacias84
Champion

I seen this problem before. The problem is not with your delim, but with your field names in FIELDS. Change your "-" to "_" or remove "-" entirely. Splunk tends to not like the dash when naming fields.

Example changes:

  • cs-host to cs_host
  • cs(User-Agent) to cs_userAgent . Personally I typically use all lower case.
  • cs(Referer) to cs_referer

Doing this should fix your problem.

Also this simplified props file that works.


[iis_ex]
pulldown_type=true
MAX_TIMESTAMP_LOOKAHEAD=19
TIME_FORMAT=%Y-%m-%d %H:%M:%S
SHOULD_LINEMERGE=false
CHECK_FOR_HEADER=false
TZ=GMT
REPORT-iisw3cfields=iisw3cfields

Hope this help or gets you started cheers.

0 Karma

linu1988
Champion

i think it's overcooked. Let it flow into splunk. Splunk should be able to get the events divided whereever it finds time at the start. After that, do the field extraction on the UI, which is easier, where you can actually see the matching. If the time is not correct, set the Props.conf for time recognition. Thanks

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...