Heavy forwarder not forwarding IIS logs, but other logs fine

[eblackburn]

We are forwarding IIS logs from UFs to a heavy forwarder, and the heavy forwarder is supposed to send them on to a 3rd party. I've confirmed using packet captures that the UFs are getting the logs to the heavy forwarder, but for some reason, the HF isn't sending them on. It is, however, sending other data sources I've configured, like WinEventLogs and DHCP logs.

Does anyone know what might be causing those specific logs to stall at the HF? I'm really stumped by this one. There is one caveat, further below, where we can get it to work, but it's not optimal.

Here's basically how we have it configured:
UFs clone all data by using two tcpout groups: (1) send to indexers and (2) send to HF

HFs does this:

[indexAndForwarder] is set to "false"

Supposed to send only WinEventLog, DHCP logs, and IIS logs, filtering out everything else

## props.conf ##

[source::WinEventLog:*] - this works

TRANSFORMS-routing = 3rdpartyOut

[DhcpSrvLog] - this works

TRANSFORMS-routing = 3rdpartyOut

[ms:iis:auto] - does not work - also tried using the source instead of using sourcetype
TRANSFORMS-routing = 3rdpartyOut

## transforms.conf ##

[3rdpartyOut]

REGEX = .

SOURCE_KEY = MetaData:Host
DEST_KEY = _SYSLOG_ROUTING
FORMAT = 3rdparty

## outputs.conf ##

# Defaults - routes everything to "nothing" by default
[syslog]
defaultGroup=nothing

[syslog:3rdparty]
sendCookedData=false
server = x.x.x.x:xxxx

Couple of random notes:

- We are using a separate transforms and props to manually tag all IIS logs with "IISWebLog" (thanks to someone on this forum for help with that)
- If we actually use the inputs.conf on the TA we built for UFs, that tells those UFs to clone their data for the heavy forwarder, to start monitoring IIS logs (in other words, not changing anything on the HF), it actually works - the logs get ingested to Splunk, and sent all the way to the 3rd party; but for some reason, they don't get the "IISWebLog" tag. This also gets pushed out to way more servers than we actually want to monitor IIS for, so this wouldn't be ideal anyway. But it's interesting that it somehow gets the logs all the way to the end.

Thank you for any help!

[scelikok]

Hi @eblackburn,

I am not sure but I think it worths to try. You can find more information about IIS sourcetypes below;

https://docs.splunk.com/Documentation/AddOns/released/MSIIS/Sourcetypes

[scelikok · Accepted solution]

Hi @eblackburn,

Did you try with "ms:iis:default" sourcetype? Since "ms:iis:auto" is special sourcetype that force Splunk for indexed time extractions, maybe that is why Heavy Forwarder cannot make transform.

Of course this may makes you check/update your search extraction settings on Splunk search head.

[eblackburn]

Hey @scelikok, thanks for your reponse! That's interesting. The way we have our IIS TA configured, there's only an inputs.conf in the /local folder, and it only uses the auto sourcetype:

[monitor://C:\inetpub\logs\LogFiles\*.log]

sourcetype = ms:iis:auto

So that's the only sourcetype we have in Splunk as well. Do you think that if we changed to using the ms:iis:default sourcetype at the UF, the heavy forwarder would be able to transform it and send it out? If so, I may give it a try and try to figure out what changes we'd need to make on the SHs.