All Apps and Add-ons

Splunk Add-on for Microsoft IIS: IIS logs are a complete mess. Is this due to user error or the way Splunk is indexing the data?

Explorer

It could just be me and my configuration, but somethings is amiss.

on my iis server I have:

[monitor://D:\Logs\WebLogs\*\*.log]
disabled = false
sourcetype = ms:iis:auto
ignoreOlderThan = 14d
index = iis1

and I have installed "Splunk Add-on for Microsoft IIS" v1.0

From, there I thought is was pretty straight forward, if using w3c and auto, with the latest version of Splunk, it should be able to parse and index correctly, but it is not.

When I do a basic search, looking at the index, I have a number of entries called fieldx(1,2,3,4,...) as well as EXTRA_FIELD_X

these fields, I can live with, the real issue is with the data indexing.

For instance:

under c_host, I have HTTP/1.1
under c_ip the only value is 80
dest_ip has HTTP/1.1
s_port shows /ActiveEfficiency/Devices/

this goes on and on, making the data fairly useless. While it is there and all, it is difficult to get any function out of the dashboards as "Activity by HTTP Method" shows me server names.

So, the question would be, is the issue the way Splunk is processing/indexing the data or the way the forwarder is presenting it to the Splunk server?

Hopefully this is something simple to fix and will not require custom transforms and props.conf files as that would defeat the purpose of the advertised improved iis indexing ability of Splunk. Or, maybe I completely misjudged things and am expecting something that is not.

thanks

0 Karma

Builder

So I forgot where I got it from, but I built my own based on a blog I found. Basically it uses transforms and props to read the top line of the iis log, and get its field from that.

I did have to restart IIS -OR- change the log rotation to 1 hour so that it would generate a new logfile with the fieldnames at the top for it to work.

cs_User_Agent extracts for crap sometimes.

One last thing.. It helps greatly if all your IIS server output in the same, or close to the same, order of fields. I also usually tuck cs_user_agent at the end, so if the extractions fails for that field the ones behind it in the line do not get shifted into other fields.

good luck.

-JD

Windows_iis.zip

0 Karma

Explorer

when you say cs_User_Agent doesnt extract well at times, where is this pulling from?

Also, what are the advanced logs? I only have the w3c logs, unless of course I am looking in the wrong place. In IIS there is only the one log setting, so surely there is another spot.

0 Karma

Builder

So, in this example, The user agent is Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+6.1;+Trident/4.0)

2017-02-17 19:19:37 W3SVC2 HOSTNAMEREDACTED 10.214.4.81 GET /HRA/HRA.aspx - 80 - 172.22.101.10 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+6.1;+Trident/4.0) SavedUserName=REDACTED;+ASP.NET_SessionId=0xjpam55w1fsv255yzgiod32;+LSessID=1218817424;+SessID=1425617692;+EncSess=mKIKj8I/WJQZOOBBr+WcXA https://URLREDACTED 200 0 0 126905 530 171

IIS Advanced logging is an add-on for IIS that allows capture of things like X-For-Forwarded IP addresses and a few other custom fields. https://www.iis.net/downloads/microsoft/advanced-logging

0 Karma

Explorer

OK, makes a bit more sense now. One question, is IIS advanced the same as IIS only better? Meaning, I am able to extract the same thing with advanced as I can with basic IIS logging. However, advanced can the extract more.

I guess what I am asking is, do I need both for the same server? I assume no as advanced allows the re-ordering as you suggest.

I would think the best practice would be to always move commonly empty fields to the end?

Lastly, when it comes to IIS logging, is there stuff that is just garbage (not needed) vs the must have fields to log? Thanks!

0 Karma

Builder

IIS Advanced logging will let you view all the same fields. But unless you have a need for X-Forwarded-For (Behind a load balancer, and want to know the actual IP Address the client was from) there isn't much need in my experience to install it.

You can change the order of the fields under logging options per-site, or globally.

As far as junk fields, that is really up to you.

Here's a list.
https://technet.microsoft.com/en-us/library/cc754702%28v=ws.10%29.aspx?f=255&MSPPError=-2147217396

From this list, the only ones that I rarely use are s-port,and maybe cookie.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!