It could just be me and my configuration, but somethings is amiss.
on my iis server I have:
[monitor://D:\Logs\WebLogs\*\*.log] disabled = false sourcetype = ms:iis:auto ignoreOlderThan = 14d index = iis1
and I have installed "Splunk Add-on for Microsoft IIS" v1.0
From, there I thought is was pretty straight forward, if using w3c and auto, with the latest version of Splunk, it should be able to parse and index correctly, but it is not.
When I do a basic search, looking at the index, I have a number of entries called fieldx(1,2,3,4,...) as well as EXTRAFIELDX
these fields, I can live with, the real issue is with the data indexing.
under c_host, I have HTTP/1.1 under c_ip the only value is 80 dest_ip has HTTP/1.1 s_port shows /ActiveEfficiency/Devices/
this goes on and on, making the data fairly useless. While it is there and all, it is difficult to get any function out of the dashboards as "Activity by HTTP Method" shows me server names.
So, the question would be, is the issue the way Splunk is processing/indexing the data or the way the forwarder is presenting it to the Splunk server?
Hopefully this is something simple to fix and will not require custom transforms and props.conf files as that would defeat the purpose of the advertised improved iis indexing ability of Splunk. Or, maybe I completely misjudged things and am expecting something that is not.
So I forgot where I got it from, but I built my own based on a blog I found. Basically it uses transforms and props to read the top line of the iis log, and get its field from that.
I did have to restart IIS -OR- change the log rotation to 1 hour so that it would generate a new logfile with the fieldnames at the top for it to work.
csUserAgent extracts for crap sometimes.
One last thing.. It helps greatly if all your IIS server output in the same, or close to the same, order of fields. I also usually tuck csuseragent at the end, so if the extractions fails for that field the ones behind it in the line do not get shifted into other fields.
when you say csUserAgent doesnt extract well at times, where is this pulling from?
Also, what are the advanced logs? I only have the w3c logs, unless of course I am looking in the wrong place. In IIS there is only the one log setting, so surely there is another spot.
So, in this example, The user agent is Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+6.1;+Trident/4.0)
2017-02-17 19:19:37 W3SVC2 HOSTNAMEREDACTED 10.214.4.81 GET /HRA/HRA.aspx - 80 - 172.22.101.10 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+6.1;+Trident/4.0) SavedUserName=REDACTED;+ASP.NET_SessionId=0xjpam55w1fsv255yzgiod32;+LSessID=1218817424;+SessID=1425617692;+EncSess=mKIKj8I/WJQZOOBBr+WcXA https://URLREDACTED 200 0 0 126905 530 171
IIS Advanced logging is an add-on for IIS that allows capture of things like X-For-Forwarded IP addresses and a few other custom fields. https://www.iis.net/downloads/microsoft/advanced-logging
OK, makes a bit more sense now. One question, is IIS advanced the same as IIS only better? Meaning, I am able to extract the same thing with advanced as I can with basic IIS logging. However, advanced can the extract more.
I guess what I am asking is, do I need both for the same server? I assume no as advanced allows the re-ordering as you suggest.
I would think the best practice would be to always move commonly empty fields to the end?
Lastly, when it comes to IIS logging, is there stuff that is just garbage (not needed) vs the must have fields to log? Thanks!
IIS Advanced logging will let you view all the same fields. But unless you have a need for X-Forwarded-For (Behind a load balancer, and want to know the actual IP Address the client was from) there isn't much need in my experience to install it.
You can change the order of the fields under logging options per-site, or globally.
As far as junk fields, that is really up to you.
From this list, the only ones that I rarely use are s-port,and maybe cookie.