EDIT: I have created a feature request to remove settings that set index-time fields at search-time in hopes of saving other people from issues like this. Please feel free to support the issue: https://ideas.splunk.com/ideas/APPSID-I-99
NOTE: Other TA's even by Splunk have this bad behavior of modifying index time fields at search time, so if you are seeing similar symptoms with other logs, this answer my be helpful.
The problem is in Splunk's Microsoft TA for both sourcetypes (ms:iis:default and ms:iis:auto) they do a field alias to host.
FIELDALIAS-s_computername = s_computername as host
Problem #1 was the props.conf didn't get deployed to the universal forwarders so the there weren't indexed extractions. Since we were using ms:iis:default, there weren't search time extractions, so s_computername field had no value and thus the alias unset the host value. Deploying the props.conf fixed some of our logs. Honestly that was just an oversight on the person configuring the logs; to me the problem is overriding host because we concentrated on that since host is a very fundamental field in Splunk and we ignored the missing field extractions. Had host been working as expected, we would have said the field extractions were missing and the missing props.conf on the UF would have been the first thing I checked.
Problem #2 was deploying the props.conf didn't fix all of our logs; some of our apps still had the empty host value at search time. The problem is these particular logs don't have the s_computername field. Why should they since it is redundant with the host field?
I updated the support case saying I fixed it by disabling that field alias (we are cloud, so I can't remove or disable the setting) and requested they fix the TA. I wrote out an explanation that modifying an index-time field at search-time should always be considered a bug. If the values are the same, cycles are just wasted. If the values are different, however, odd search results occur because host:: and host= have different values. If s_computername honestly holds the correct value vs host, then its value should be populated into host at index time. That's fairly easy with ms:iis:auto since it does index time extractions; ms:iis:default,however, doesn't have a great option since s_computername isn't known until search time and part of the point of the TA's is you can modify the fields in the logs.
I suggested they:
Fix the Splunk Add-on for Microsoft IIS to remove that setting
Update the Splunk base guidlines and app inspect tool to require the index time fields be modified only at index time
Unfortunately I was just told that the behavior of alias changed in 7.3.x and their only suggestion is to open feature requests. I intend to open the feature request (and update this so other can vote for the fix), but the site has been returning 503 errors. Since the feature request will likely get drowned in the sea of noise and updates take awhile, I wanted to post here to hopefully save other people some headache.
... View more