I have been reading through the Web Analytics documentation and the third point under 'Searches aren't returning any data" says:
File field not present
Another field that is known to cause problems is the "file" field. This needs to be present in your field extractions and if it is not, you will not see the "eventtype-=pageview" which is necessary for the app to work. Make sure this is extracted correctly.
Now, I have IIS logs (W3SVC format which is standard IIS), and there is no 'File' field present, nor can it be added in IIS Manager. by using 'Select Fields' under Logging. So, how can IIS users get this field, and hence, get Web Analytics to work with IIS? Thanks.
New version of the app is now live which hopefully solve this issue.
- Added an option to use a different data model name than "Web". This caused conflicts with the default CIM datamodel also called Web.
- Made changes to Sites setup dashboard to make it easier.
- Migrated website setup settings to the KV store.
- Added better support for IIS. Now supports ms:iis:auto and ms:iis:default sourcetypes which comes from the official IIS Add-on.
- Updated User agent string parsing to latest version
- Various bug fixes
I think you are misunderstanding the concept of the file field in relation to this app. This field is a search time extraction that happens inside Splunk. There is no need to re-configure the iis server logs to include this field.
The file field comes from an extraction based on the cs_uri_stem field which contains the requested resource including the file.
iis : EXTRACT-file
-spk12 SplunkAppForWebAnalytics]# grep -iR cs_uri_stem
default/props.conf:EXTRACT-file = (?\w+(?:.\w+)+$) in cs_uri_stem
Right. OK, Thanks. This is in relation to getting the Web Analytics App to work with IIS logs, which default to W3C. No one running web stuff on IIS has been using the 'IIS' log format for quite some time .... most likely because everyone knows that Apache is a much better web server, but hey, some of us work for organizations that are structured around the Evil Empire of MS so a Sith engineered solution is the only option. But it is telling that Microsoft itself changed from its very poor IIS log format to W3C, because even they knew their proprietary format was really really bad. I wonder why they just didn't take the old 'Embrace, Extend, Extinguish' page from the playbook and switch to an Apache log format so we could all just use access_combined. But that would make too much sense, and make everyone's life too easy, wouldn't it? But I digress.
The author of the Web Analytics app got back to me on another thread, and the short answer is to use the Microsoft IIS Add-On along with a sourcetype of ms:iis:auto and change the props.conf file for the mappings to look sensible (e.g. Apache logs). Do I want to throw IIS out the window? Yes, for a long time. Can I? Unfortunately not.
That is so awesome, I went to great lengths to switch from ms:iis:auto to basic iis.[I'm gonna rename it ms:iis:EXTRA!!!!]
Let me see if i can one shot the test data before I scrub the index from the cluster, again....
Hopefully just pretty much add: sourcetype="ms:iis:auto"?
[ok yeah confirmed in that other thread you mentioned above]
Name Search string Tag(s)
web-traffic sourcetype="aws:cloudfront:accesslogs" OR sourcetype="apache:access" OR sourcetype="iis" OR sourcetype="access_combined" OR sourcetype="access_common" OR sourcetype="access_combined_wcookie" OR sourcetype="ms:iis:auto"
Yuhp so far in dev a one shot of the sample data worked;
[thank you for speaking up][I saw all the chatter about other sourcetypes but didn't recognize when the author was speaking]
[a bit less than intuitive otherwise]
[I was like forget it, this time I am following what the author says to the T! basic vanilla iis all the way! No.]
[Shame on me I should have known; The Dark Side of the Force is the pathway to many abilities some consider to be… Unnatural.][that's all I got]
Leveraging:Splunk Add-on for Microsoft IIS
author = Splunk
version = 1.0.0 [I am not going to mess with 1.0.1 if 1.0.0 works]
description = Splunk Add-on for Microsoft IIS
[hopefully everything will work out with the rest of my sharepoint farms, these boys want to machine learn their data real bad]
Reach out to me via email and I can help you out. It's my firstname and then splunk.com. Simple
Just doing some development work on the app right now.
OK, Will Do, Johan, I think I am on the right track switching this logs over to sourcetype ms:iis:auto, and I will definitely engage your offer if this does not work, I'll try not to bother you if this pans out..
I didn't use the Web Analytics app, but I've gotten IIS logs to ingest before. As I recall when the service starts or the log rotates the new file starts with a header line that begins with "#Fields:"
As seen here.
Anyways, here are my inputs and props.conf that I just bundled up into a deployment app. They still work for me
[monitor://C:\inetpub\logs\LogFiles\...\] disabled = 0 whitelist = .log$ followTail = 0 sourcetype=iis ignoreOlderThan = 15d index = iis [monitor://E:\inetpub\logs\LogFiles\...\] disabled = 0 whitelist = .log$ followTail = 0 sourcetype=iis ignoreOlderThan = 15d index = iis
[iis] ... INDEXED_EXTRACTIONS = w3c FIELD_DELIMITER = whitespace FIELD_HEADER_REGEX = ^#Fields:\s*(.*) MISSING_VALUE_REGEX = - TIME_FORMAT = %Y-%m-%d %H:%M:%S TIMESTAMP_FIELDS = date,time TRANSFORMS-set= setnull
Transforms.conf - that uses setnull to weed out loadbalancer web hits from the logs.
# This stanza removes the loadbalancer noise. [setnull] REGEX = _hc_.txt DEST_KEY = queue FORMAT = nullQueue
Thanks for that. I can get Splunk to ingest iis logs. One of the issues that I discovered was that initCrcLength is required along with alwaysOpenFile, otherwise Splunk will not ingest all the logs, only a few
[monitor://C:\inetpub\logs\LogFiles\W3SVC1\*] index = weblogs disabled = false sourcetype = iis ignoreOlderThan = 90d initCrcLength = 2310 alwaysOpenFile = 1
The developer seems to have answered my question in another thread here: https://answers.splunk.com/answers/727931/is-it-easy-to-ingest-advanced-iis-logs-into-the-sp.html and it does seem to be an issue where IIS has an 'iis' log type, but the 'w3svc' log type is now (and has been for a while) the default iis log type. So sourcetype=iis is pretty misleading nowadays since that should default to w3svc formatted iis logs these days and not 'iis formatted' which Microsoft itself seems to have quietly shoved into the corner with a little Dunce cap.