Getting Data In

Why am I getting the IP address details instead of the port number for iis logs?

Motivator

Hi All,

We are facing a parsing issue for iis logs and the issue is that there is only for few of the host not on all the hosts which are ingesting the iis data in Splunk.

When we search index=web sourcetype=iis:web:common host=test01 we could see that in the interesting filed under s_port we are getting the IP address details instead of the port number. However, this isn't consistent across all devices.

Kindly guide me how/ where to start my investigation.

Thanks in advance.

Tags (2)
0 Karma

SplunkTrust
SplunkTrust

You have already made the biggest advance in the investigation by realizing that some hosts extract correctly and others don't. There are several "formats" IIS logs can be written in, and as many custom formats as you can think of.

Your easiest solution would be to carefully check the IIS logging configuration on one that doesn't work right and compare it with one that does. This should find the problem. And, don't forget that logging configuration can be overridden on a site-per-site basis, so you may have to check not only the main configuration, but how each site is configured.

If you confirm those configurations are exactly the same, then you'll have to confirm the next piece toward Splunk - presuming you are using a UF on the host to collect the files, double-check the inputs settings are the same between the two.

If you still haven't found anything, then check the indexers and SHs. But I think you'll find it before this point.

Oh, and if you really can't find it even with all that, paste into here an event from each - one that works and one that doesn't - and if you can find it the extraction that's being done. We'd be happy to help!

Happy Splunking,
-Rich

0 Karma

Builder

Are the field autoextracted or you have written field extractions for iis:web:common?

0 Karma

Motivator

hi its not autoextracted, as i could see Report, Extract stanza along with Field alias has been used in props.conf.

0 Karma

Motivator

Hi rich, could you please guide me on how to check the IIS logging configuration details, as I don't have much knowledge on IIS configuration.

Yes we are using UF to collect the IIS log data and this is parsed in HF and then get indexed. There are 42 remote IIS node machine ingesting the iis log data in splunk and all are sharing the same inputs.conf detail, as the configuration is managed and push centrally via deployment server.

Inputs.conf details:

[monitor://C:\Windows\inetpub\logs\LogFiles\W3SVC*\*.log]
ignoreOlderThan = 3d
index=web
sourcetype=iis:web:common

[monitor://D:\LogFiles\W3SVC*\*.log]
ignoreOlderThan = 3d
index=web
sourcetype=iis:web:common

[monitor://E:\inetpub\logs\LogFiles\W3SVC*\*.log]
ignoreOlderThan = 3d
index=web
sourcetype=iis:web:common

Kindly let me know how to fix this issue.

0 Karma

SplunkTrust
SplunkTrust

Hemnaath,

MS has instructions for checking or changing the logging levels and style in IIS.

@ssadanala1 also makes a great point - How are you parsing those log files? There IS an app (actually more than one if I recall correctly) that handles much of this. I figured you were having a "off by one" field problem on some servers, which may still be the case, but maybe it's that you are trying to roll your own IIS extractions?

Either way, can you please check the IIS log settings - you are looking for variations on WHAT is being logged, check how you are parsing the logs (and is that the right sourcetype for the app mentioned to use), then lastly if no answer seems to pop out with those, please paste in one event that's right and one event that's wrong so we can see the EVENTS that are having an issue and compare it with one that isn't?

0 Karma

Motivator

Hi rich, Please find the Event data

Event Details: Which is working fine.

2018-02-01 11:28:18 W3SVC1 test01 10.x.x.7 POST /XMII/Illuminator QueryTemplate=AIM%2FCommon%2FGetLineShiftXacute&Content-Type=text%2Fxml&X-ARR-CACHE-HIT=0&X-ARR-LOG-ID=4d2ca237-5c38-4436-bd34-d74c8c345cda 80 - 10.x.x.157 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.1;+WOW64;+Trident/7.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+.NET4.0C;+.NET4.0E) ACTSESSION=aHR0cDovL3N0dWFydHNkcmxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx1hNSUkvQ00vQUlNL1BhY2thZ2luZy9Db21wbGV0ZVF1YWxpdHkuaXJwdA%3D%3D;+CTSESSION=AAAAAgABAGBVh14WQ8UkIIg54kAqRirk8rjD2c9%2Ffu%2BpeYSWtWpsd7kYF5NX6s6EE58Atx9yk6xRLY3je6i8c3Xav5LmGAqnupZhuremKw8XWJDDGuYLFv86vSVVkLulZA9V%2BeshH%2Bw%3D;+ARRAffinity=8a9bfbe1cddd7747c4cf1be435c17290c2c92477166719b6265b2b5d8634f683;+saplb_*=(J2EE7745720)7745751;+JSESSIONID=wt4-0D-5TBFLpqthMEyB4OlXoRIoYQHXMHYA_SAP5BJ1xzFg6IdsGx1xGGXXNk7r;+JSESSIONMARKID=IEy6wwbT5obwlmyqCHvHEur-beZ_QrsSK05NcwdgA;+MYSAPSSO2=AjExMDAgABJwb3J0YWw6U1RVTDQ3T1BSMDKIAAdkZWZhdWx0AQALU1RVTDQ3T1BSMDICAAMwMDADAANNMDcEAAwyMDE4MDIwMTA0MDAFAAQAAAAICgALU1RVTDQ3T1BSMDL%2FAQUwggEBBgkqhkiG9w0BBwKggfMwgfACAQExCzAJBgUrDgMCGgUAMAsGCSqGSIb3DQEHATGB0DCBzQIBATAiMB0xDDAKBgNVBAMTA00wNzENMAsGA1UECxMESjJFRQIBADAJBgUrDgMCGgUAoF0wGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTgwMjAxMDQwMDE1WjAjBgkqhkiG9w0BCQQxFgQUpXQoBd7SqP%2FgEyCwD!w5T3NAGKMwCQYHKoZIzjgEAwQvMC0CFDRfXhDnQmcZYDpDVcOrbHZZ5WZSAhUAxQXo2IZtxEGicAdDy3gmCMmm9G8%3D http://stard.aim.xxxx.com/XMII/CM/AIM/Packaging/CompleteQuality.irpt star.aim.xxxx.com 200 0 0 712 1716 109

Event Detail: Which is not working correctly

2018-01-31 13:45:07 10.x.x.x GET /areas/ehs/book_style/erp_learning_book/textbox/textbox.css - 80 - 10.x.x.245 Mozilla/5.0+(Windows+NT+6.1;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/63.0.3239.132+Safari/537.36 http://erphelp.xxxx.com/areas/ehs/trainer/10_x_x_1406/slide.htm 304 0 0 218

Kindly guide me how/where to trouble shoot to fix this issue.

0 Karma

SplunkTrust
SplunkTrust

Right, those two events don't line up. If you just put them both into notepad on two lines and use the tab key on all the spaces until each field lines up, you'll see they're different. There's two extra fields in one of them. You can see that in just the first half dozen fields:

2018-02-01 11:28:18     W3SVC1      test01  10.x.x.7    POST
2018-01-31 13:45:07                         10.x.x.x    GET 

Now, how did they get that way? Again I refer back to your IIS settings. I linked before on how to check those, but it appears these two systems are using different settings. Open up IIS on both of them, side by side. Look at the settings as per the docs linked, see where they differ.

Another point though - IIRC IIS logs HAVE A HEADER LINE and I believe the IIS add-on uses this to make the fields correct, then just aliases things to make them match in certain cases. It may be useful to install and configure the app, and use that as the sourcetype and see if this doesn't get resolved from that.

0 Karma

Motivator

Hi rich, thanks for your effort on this, hey I am not familiar with the IIS settings so can you please guide me on this or share me the doc link to validate the configuration details.

Another point though - IIRC IIS logs HAVE A HEADER LINE and I believe the IIS add-on uses this to make the fields correct, then just aliases things to make them match in certain cases. It may be useful to install and configure the app, and use that as the sourcetype and see if this doesn't get resolved from that.

I did not understand this comment, when checked the splunk add-on for iis, could not come across IIRC IIS logs, so can you help me on this too.

Hey can I paste the props.conf and transforms.conf detail in the comment, so that you can guide me where to correct the stanza to fix the issue, will this be better idea.

thanks in advance.

0 Karma

SplunkTrust
SplunkTrust

IIRC is "If I recall correctly". 🙂

MS has instructions for checking or changing the logging levels and style in IIS. Regardless of my next instructions about the app, this step needs done. Again I don't know exactly what you are looking for in those configurations - they're MS IIS configuration options on the Web server, not a Splunk thing. But as I mentioned if you just open up the configuration utility once on a server "that works" and once on a server "that doesn't work" and follow MS's guide on where to look on each, you will likely find a difference. In any case, according to the section of the docs for setting up the sourcetype, you will want them all to be set to "W3C format".

While you could put the props and transforms here, I'm going to suggest that you instead install the Splunk Add-on for Microsoft IIS and get your inputs and sourcetypes configured so that your IIS logs will use this. There's ample documentation - take a read through it and give this a try. It will solve this problem and many other problems beside.

(Now, one thing to note is if you have created a lot of work around your own extractions, the fieldnames will all be different now so that might need fixing on any reports you've already done. But the payoff will be great - not having to maintain your own extractions, not having to fix all the little corner cases you find, and having it be consistent across all your logs - is priceless!)

0 Karma

Contributor

There is an app out there in splunk base for IIS logs

https://splunkbase.splunk.com/app/3185/