All Apps and Add-ons

Is it easy to ingest advanced IIS Logs into the Splunk App for Web Analytics?

ksoori
New Member

Can we easily ingest advanced IIS Logs into the Splunk App for Web Analytics? Does this app support by default?

Does it require some customization?

This app works fine for normal IIS logs and i tried configuring advanced IIS logs. I got to show green tick mark under "Website Setup" but none of the dashboard panel gets updated with any data.

Any advice on this would be helpful.

0 Karma

jbjerke_splunk
Splunk Employee
Splunk Employee

Hi

New version of the app is now live which hopefully solve this issue.
https://splunkbase.splunk.com/app/2699

v 2.2.0
- Added an option to use a different data model name than "Web". This caused conflicts with the default CIM datamodel also called Web.
- Made changes to Sites setup dashboard to make it easier.
- Migrated website setup settings to the KV store.
- Added better support for IIS. Now supports ms:iis:auto and ms:iis:default sourcetypes which comes from the official IIS Add-on.
- Updated User agent string parsing to latest version
- Various bug fixes

0 Karma

ksoori
New Member

i was able to make it work by copying [iis] in the props.conf file and making a new version for the advanced logging.

[adv_iis]

0 Karma

ksoori
New Member

i was able to make this work by making a copy of the [iis] section in the props.conf and naming it as [adv_iis] . I also updated the new fields that pops out on the advanced version.

It now works like a charm

0 Karma

kmower
Communicator

I am wondering the same thing - specifically, will it work with the splunk IIS add-on for the sourcetype ms:iis:auto which I am using for W3SVC formatted logs in IIS.

0 Karma

jbjerke_splunk
Splunk Employee
Splunk Employee

Hi

I am working on support for this sourcetype and it will be available in the next version.

In the meantime I have this props.conf stanza available.

[ms:iis:auto]
CHARSET=UTF-8
INDEXED_EXTRACTIONS=w3c
MAX_TIMESTAMP_LOOKAHEAD=32
SHOULD_LINEMERGE=false
category=Web
description=W3C Extended log format produced by the Microsoft Internet Information Services (IIS) web server
detect_trailing_nulls=auto
pulldown_type=true

EXTRACT-http_referer_domain = https?:\/\/(?<http_referer_domain>[^/]+) in cs_Referer
EVAL-http_referer = if(isnull(cs_Referer),"-",cs_Referer)
FIELDALIAS-clientip = c_ip AS clientip
FIELDALIAS-cookie = cs_Cookie AS cookie
FIELDALIAS-http_user_agent = cs_User_Agent AS http_user_agent
FIELDALIAS-bytes = cs_bytes AS bytes
#FIELDALIAS-host = cs_host AS host
EVAL-host = coalesce(cs_host,cs_Host,host)
FIELDALIAS-http_method = cs_method AS http_method
FIELDALIAS-uri_query = cs_uri_query AS uri_query
FIELDALIAS-cs_uri_stem = cs_uri_stem AS uri
FIELDALIAS-uri = cs_uri_stem AS http_request
FIELDALIAS-user = cs_username AS user
FIELDALIAS-version = cs_version AS version
FIELDALIAS-status = sc_status AS status
FIELDALIAS-response_time = time_taken AS response_time
#EXTRACT-file = .*[/](?<file>.+\.\w+) in cs_uri_stem
EXTRACT-file = (?<file>\w+(?:\.\w+)+$) in cs_uri_stem

#Global properties, applied to all sourcetypes for the app
EXTRACT-http_locale = (?i)^(?:[^;\n]*;){3}\s+(?P<http_locale>[a-z]{2}(|[-_][a-z]{2}));
EVAL-file = if(match(file,"\."),file,NULL)
EVAL-http_channel = if(http_referer="-","Direct", if(like(http_referer_domain,"%".site."%","Direct", if(isnull(http_channel), "Referal", http_channel)))
EVAL-http_referer_domain = replace(http_referer_domain, "http(s|):\/\/", "")
EVAL-http_referer_hostname = replace(replace(replace(http_referer_domain, "http(s|):\/\/", ""), "^(www|m|uk|r|l|tpc|lm)\.+", ""), "(\.{1}[a-zA-Z]+)", "")
EVAL-user = md5(clientip."_".http_user_agent)
LOOKUP-2_Channels = WA_channels Hostname AS http_referer_hostname OUTPUT Channel AS http_channel
LOOKUP-site = WA_settings source AS source host AS host OUTPUTNEW value AS site

You also need to modify the eventtypes.conf to refefence this sourcetype

[web-traffic]
search = sourcetype="aws:cloudfront:accesslogs" OR sourcetype="apache:access" OR sourcetype="iis" OR sourcetype="access_combined" OR sourcetype="access_common" OR sourcetype="access_combined_wcookie" OR sourcetype="ms:iis:auto"
0 Karma

kmower
Communicator

I have not had any luck with generating pages. I have been able to generate sessions, but generating pages is still a no-go for sourcetype [ms:iis:auto]. I think it may have something to do with http_request and how it is defined in props.conf (or more specifically, not defined). Any help appreciated, I feel like I am very close, just one tweeked setting away ... I just wish I knew what that setting was. Thanks.

0 Karma

kmower
Communicator

I have added the [ms:iis:auto] stanza to props.conf within the Web Analytics app, and I also added the [web-traffic] stanza to eventtypes.conf in etc\apps\SplunkAppForWebAnalytics\local

I then restarted Splunk. I tried to generate sessions again (got some) and pages (but nothing there), and the Web Analytics app still doesn't seem to be displaying anything of value.

I wonder if I need to start from a clean slate, and if so, do I need to delete anything other than the app (lookups etc.) ?

0 Karma

kmower
Communicator

That's great and music to my ears (eyes). And the W3SVC log format, which is now the default source type in IIS? I have switched to this since it doesn't seem like I should need the IIS Add-On or ms:iis:auto . That is, by default iis uses the w3svc format so sourcetype=iis should be synonymous with iis these days, unless the user goes into IIS and changes this back to the old 'iis' log format.

That is, could I just use that props.conf file for the iis sourcetype if/when using w3svc? Thanks again.

0 Karma

jbjerke_splunk
Splunk Employee
Splunk Employee

Hi

You should be able to use the props above for any sourcetype name, iis or ms:iis:auto. Just make sure the naming lines up between the actual logs, the inputs, the props and the eventtypes.

I'm the author of the app and I will improve the support in the next version. We are looking at a May 2019 release. Sorry for the iis neglect!

j

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...