All Apps and Add-ons

IIS Logs: Missing host field / inconsistent host values ms:iis:default and ms:iis:auto

triest
Communicator

When searching IIS logs, there is some irregularity on the host field.

Originally this was reporting as IIS logs where missing but that the OS logs where showing up as expected.

When I originally when to investigate this, I picked a particular server and did a tstats and I saw both the OS and IIS logs. | tstats count where host=pdwww1 by index sourcetype

Since I saw the IIS logs in tstats, I went to pull them up ( index=iis host=pdwww1) and received no results.

I then did a generic index search and the logs where there, but I noticed there was no host field. That's really odd since the operating system logs are showing up as expected and the host value for those logs and the IIS logs is from the configuration merging of the host value in $SPLUNK_HOME/etc/system/local.inputs.conf

I then tried doing a search using the index-time style index=iis host::pdwww1 which returned the logs as expected. So we can find the logs, but doing things like index=iis | stats count by host doesn't work correctly due to the missing host value at search time.

0 Karma
1 Solution

triest
Communicator

EDIT: I have created a feature request to remove settings that set index-time fields at search-time in hopes of saving other people from issues like this. Please feel free to support the issue: https://ideas.splunk.com/ideas/APPSID-I-99

NOTE: Other TA's even by Splunk have this bad behavior of modifying index time fields at search time, so if you are seeing similar symptoms with other logs, this answer my be helpful.

The problem is in Splunk's Microsoft TA for both sourcetypes (ms:iis:default and ms:iis:auto) they do a field alias to host.

FIELDALIAS-s_computername = s_computername as host

Problem #1 was the props.conf didn't get deployed to the universal forwarders so the there weren't indexed extractions. Since we were using ms:iis:default, there weren't search time extractions, so s_computername field had no value and thus the alias unset the host value. Deploying the props.conf fixed some of our logs. Honestly that was just an oversight on the person configuring the logs; to me the problem is overriding host because we concentrated on that since host is a very fundamental field in Splunk and we ignored the missing field extractions. Had host been working as expected, we would have said the field extractions were missing and the missing props.conf on the UF would have been the first thing I checked.

Problem #2 was deploying the props.conf didn't fix all of our logs; some of our apps still had the empty host value at search time. The problem is these particular logs don't have the s_computername field. Why should they since it is redundant with the host field?

I updated the support case saying I fixed it by disabling that field alias (we are cloud, so I can't remove or disable the setting) and requested they fix the TA. I wrote out an explanation that modifying an index-time field at search-time should always be considered a bug. If the values are the same, cycles are just wasted. If the values are different, however, odd search results occur because host:: and host= have different values. If s_computername honestly holds the correct value vs host, then its value should be populated into host at index time. That's fairly easy with ms:iis:auto since it does index time extractions; ms:iis:default,however, doesn't have a great option since s_computername isn't known until search time and part of the point of the TA's is you can modify the fields in the logs.

I suggested they:

  1. Fix the Splunk Add-on for Microsoft IIS to remove that setting
  2. Update the Splunk base guidlines and app inspect tool to require the index time fields be modified only at index time

Unfortunately I was just told that the behavior of alias changed in 7.3.x and their only suggestion is to open feature requests. I intend to open the feature request (and update this so other can vote for the fix), but the site has been returning 503 errors. Since the feature request will likely get drowned in the sea of noise and updates take awhile, I wanted to post here to hopefully save other people some headache.

View solution in original post

triest
Communicator

EDIT: I have created a feature request to remove settings that set index-time fields at search-time in hopes of saving other people from issues like this. Please feel free to support the issue: https://ideas.splunk.com/ideas/APPSID-I-99

NOTE: Other TA's even by Splunk have this bad behavior of modifying index time fields at search time, so if you are seeing similar symptoms with other logs, this answer my be helpful.

The problem is in Splunk's Microsoft TA for both sourcetypes (ms:iis:default and ms:iis:auto) they do a field alias to host.

FIELDALIAS-s_computername = s_computername as host

Problem #1 was the props.conf didn't get deployed to the universal forwarders so the there weren't indexed extractions. Since we were using ms:iis:default, there weren't search time extractions, so s_computername field had no value and thus the alias unset the host value. Deploying the props.conf fixed some of our logs. Honestly that was just an oversight on the person configuring the logs; to me the problem is overriding host because we concentrated on that since host is a very fundamental field in Splunk and we ignored the missing field extractions. Had host been working as expected, we would have said the field extractions were missing and the missing props.conf on the UF would have been the first thing I checked.

Problem #2 was deploying the props.conf didn't fix all of our logs; some of our apps still had the empty host value at search time. The problem is these particular logs don't have the s_computername field. Why should they since it is redundant with the host field?

I updated the support case saying I fixed it by disabling that field alias (we are cloud, so I can't remove or disable the setting) and requested they fix the TA. I wrote out an explanation that modifying an index-time field at search-time should always be considered a bug. If the values are the same, cycles are just wasted. If the values are different, however, odd search results occur because host:: and host= have different values. If s_computername honestly holds the correct value vs host, then its value should be populated into host at index time. That's fairly easy with ms:iis:auto since it does index time extractions; ms:iis:default,however, doesn't have a great option since s_computername isn't known until search time and part of the point of the TA's is you can modify the fields in the logs.

I suggested they:

  1. Fix the Splunk Add-on for Microsoft IIS to remove that setting
  2. Update the Splunk base guidlines and app inspect tool to require the index time fields be modified only at index time

Unfortunately I was just told that the behavior of alias changed in 7.3.x and their only suggestion is to open feature requests. I intend to open the feature request (and update this so other can vote for the fix), but the site has been returning 503 errors. Since the feature request will likely get drowned in the sea of noise and updates take awhile, I wanted to post here to hopefully save other people some headache.

mauricio_sandov
Explorer

Any update on this. I am having same problem. I was breaking my head for days.  I confirmed same log behavior and same index/search field extraction problem. This is causing issues with apps like webayalytics.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...