All Apps and Add-ons

No file field in IIS logs

kmower
Communicator

I have been reading through the Web Analytics documentation and the third point under 'Searches aren't returning any data" says:

File field not present
Another field that is known to cause problems is the "file" field. This needs to be present in your field extractions and if it is not, you will not see the "eventtype-=pageview" which is necessary for the app to work. Make sure this is extracted correctly.

Now, I have IIS logs (W3SVC format which is standard IIS), and there is no 'File' field present, nor can it be added in IIS Manager. by using 'Select Fields' under Logging. So, how can IIS users get this field, and hence, get Web Analytics to work with IIS? Thanks.

0 Karma

jbjerke_splunk
Splunk Employee
Splunk Employee

Hi

New version of the app is now live which hopefully solve this issue.
https://splunkbase.splunk.com/app/2699

v 2.2.0
- Added an option to use a different data model name than "Web". This caused conflicts with the default CIM datamodel also called Web.
- Made changes to Sites setup dashboard to make it easier.
- Migrated website setup settings to the KV store.
- Added better support for IIS. Now supports ms:iis:auto and ms:iis:default sourcetypes which comes from the official IIS Add-on.
- Updated User agent string parsing to latest version
- Various bug fixes

0 Karma

jbjerke_splunk
Splunk Employee
Splunk Employee

Hi kmower

I think you are misunderstanding the concept of the file field in relation to this app. This field is a search time extraction that happens inside Splunk. There is no need to re-configure the iis server logs to include this field.

The file field comes from an extraction based on the cs_uri_stem field which contains the requested resource including the file.

j

0 Karma

GDustin
Path Finder

TL;DR;
Settings>Fields>Field extractions
iis : EXTRACT-file

Inline

(?\w+(?:.\w+)+$) in cs_uri_stem

No owner
SplunkAppForWebAnalytics

/opt/splunk/etc/apps/SplunkAppForWebAnalytics
-spk12 SplunkAppForWebAnalytics]# grep -iR cs_uri_stem
default/props.conf:EXTRACT-file = (?\w+(?:.\w+)+$) in cs_uri_stem

0 Karma

kmower
Communicator

Right. OK, Thanks. This is in relation to getting the Web Analytics App to work with IIS logs, which default to W3C. No one running web stuff on IIS has been using the 'IIS' log format for quite some time .... most likely because everyone knows that Apache is a much better web server, but hey, some of us work for organizations that are structured around the Evil Empire of MS so a Sith engineered solution is the only option. But it is telling that Microsoft itself changed from its very poor IIS log format to W3C, because even they knew their proprietary format was really really bad. I wonder why they just didn't take the old 'Embrace, Extend, Extinguish' page from the playbook and switch to an Apache log format so we could all just use access_combined. But that would make too much sense, and make everyone's life too easy, wouldn't it? But I digress.

The author of the Web Analytics app got back to me on another thread, and the short answer is to use the Microsoft IIS Add-On along with a sourcetype of ms:iis:auto and change the props.conf file for the mappings to look sensible (e.g. Apache logs). Do I want to throw IIS out the window? Yes, for a long time. Can I? Unfortunately not.

0 Karma

GDustin
Path Finder

That is so awesome, I went to great lengths to switch from ms:iis:auto to basic iis.[I'm gonna rename it ms:iis:EXTRA!!!!]
Let me see if i can one shot the test data before I scrub the index from the cluster, again....

Hopefully just pretty much add: sourcetype="ms:iis:auto"?
[ok yeah confirmed in that other thread you mentioned above]

Settings>Event types
Name Search string Tag(s)

web-traffic sourcetype="aws:cloudfront:accesslogs" OR sourcetype="apache:access" OR sourcetype="iis" OR sourcetype="access_combined" OR sourcetype="access_common" OR sourcetype="access_combined_wcookie" OR sourcetype="ms:iis:auto"

Yuhp so far in dev a one shot of the sample data worked;
[thank you for speaking up][I saw all the chatter about other sourcetypes but didn't recognize when the author was speaking]
[a bit less than intuitive otherwise]
[I was like forget it, this time I am following what the author says to the T! basic vanilla iis all the way! No.]
[Shame on me I should have known; The Dark Side of the Force is the pathway to many abilities some consider to be… Unnatural.][that's all I got]
ms:iis:auto
Leveraging:Splunk Add-on for Microsoft IIS
https://splunkbase.splunk.com/app/3185/#/overview
[launcher]
author = Splunk
version = 1.0.0 [I am not going to mess with 1.0.1 if 1.0.0 works]
description = Splunk Add-on for Microsoft IIS
[hopefully everything will work out with the rest of my sharepoint farms, these boys want to machine learn their data real bad]

0 Karma

jbjerke_splunk
Splunk Employee
Splunk Employee

Hi GDustin

Reach out to me via email and I can help you out. It's my firstname and then splunk.com. Simple

Just doing some development work on the app right now.

johan

0 Karma

GDustin
Path Finder

OK, Will Do, Johan, I think I am on the right track switching this logs over to sourcetype ms:iis:auto, and I will definitely engage your offer if this does not work, I'll try not to bother you if this pans out..

0 Karma

JDukeSplunk
Builder

I didn't use the Web Analytics app, but I've gotten IIS logs to ingest before. As I recall when the service starts or the log rotates the new file starts with a header line that begins with "#Fields:"
As seen here.
https://stackify.com/how-to-interpret-iis-logs/

Anyways, here are my inputs and props.conf that I just bundled up into a deployment app. They still work for me

inputs.conf

[monitor://C:\inetpub\logs\LogFiles\...\]
disabled = 0
whitelist = .log$
followTail = 0
sourcetype=iis
ignoreOlderThan = 15d
index = iis

[monitor://E:\inetpub\logs\LogFiles\...\]
disabled = 0
whitelist = .log$
followTail = 0
sourcetype=iis
ignoreOlderThan = 15d
index = iis

props.conf

 [iis]
    ...
    INDEXED_EXTRACTIONS = w3c
    FIELD_DELIMITER = whitespace
    FIELD_HEADER_REGEX = ^#Fields:\s*(.*)
    MISSING_VALUE_REGEX = -
    TIME_FORMAT = %Y-%m-%d %H:%M:%S
    TIMESTAMP_FIELDS = date,time
    TRANSFORMS-set= setnull

Transforms.conf - that uses setnull to weed out loadbalancer web hits from the logs.

# This stanza removes the loadbalancer noise.
[setnull]
REGEX = _hc_.txt
DEST_KEY = queue
FORMAT = nullQueue
0 Karma

kmower
Communicator

Thanks for that. I can get Splunk to ingest iis logs. One of the issues that I discovered was that initCrcLength is required along with alwaysOpenFile, otherwise Splunk will not ingest all the logs, only a few

[monitor://C:\inetpub\logs\LogFiles\W3SVC1\*]
index = weblogs
disabled = false
sourcetype = iis
ignoreOlderThan = 90d
initCrcLength = 2310
alwaysOpenFile = 1

The developer seems to have answered my question in another thread here: https://answers.splunk.com/answers/727931/is-it-easy-to-ingest-advanced-iis-logs-into-the-sp.html and it does seem to be an issue where IIS has an 'iis' log type, but the 'w3svc' log type is now (and has been for a while) the default iis log type. So sourcetype=iis is pretty misleading nowadays since that should default to w3svc formatted iis logs these days and not 'iis formatted' which Microsoft itself seems to have quietly shoved into the corner with a little Dunce cap.

0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

SignalFlow: What? Why? How?

What is SignalFlow? Splunk Observability Cloud’s analytics engine, SignalFlow, opens up a world of in-depth ...

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...