When trying to forward IIS logs from one indexer to another indexer, why is props.conf transform not working for the IIS stanza?

[jstacey_intuit]

From indexerA I am trying to forward Windows Event Logs and IIS Logs to indexerB. The Windows Event Logs are being forwarded properly, but the IIS Logs (sourcetype=iis) are not.

(Splunk Enterprise 6.3.3 on RHEL)

On indexerA I have the below;

///////// props.conf /////////

[iis]
CHECK_FOR_HEADER = False
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 128
TIME_FORMAT = %Y-%m-%d %H:%M:%S
TZ = UTC
REPORT-iisfields = iisfields
TRANSFORMS-routing = route2swingserver

[WinEventLog:Security]
TRANSFORMS-routing = route2swingserver

[WinEventLog:System]
TRANSFORMS-routing = route2swingserver

[WinEventLog:Application]
TRANSFORMS-routing = route2swingserver

///////// transforms.conf /////////

[route2swingserver]
REGEX=(.)
DEST_KEY=_TCP_ROUTING
FORMAT=logs4customer

///////// outputs.conf /////////

[indexAndForward]
index = true

[tcpout]
defaultGroup=nothing
maxQueueSize = 10MB

[tcpout:logs4customer]
disabled=false
server=indexerB:9998
compressed = true

[rphillips_splk · Accepted solution]

@acharlieh is correct here. If you do need to forward the iis events from indexerA to indexerB you can pass the events back to through the typingQueue with the following configurations. Here we are adding the _meta key to the events at the forwarder and then later telling splunktcp on the indexer to look for that key (_meta) in the events and route those back to the typingQueue

on UF:
$SPLUNK_HOME/etc/system/local/inputs.conf
[monitor.....youriis.log]
sourcetype=iis
_meta = iismeta

$SPLUNK_HOME/etc/system/local/default-mode.conf
[pipeline:structuredparsing]
disabled_processors=regexreplacement

on indexer:
$SPLUNK_HOME/etc/system/local/inputs.conf
[splunktcp]
route = has_key:_replicationBucketUUID:replicationQueue;has_key:_dstrx:typingQueue;has_key:_linebreaker:indexQueue;absent_key:_linebreaker:parsingQueue;has_key:_meta:typingQueue

///////props.conf//////

[iis]
CHECK_FOR_HEADER = False
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 128
TIME_FORMAT = %Y-%m-%d %H:%M:%S
TZ = UTC
REPORT-iisfields = iisfields
TRANSFORMS-routing = route2swingserver

/////transforms.conf/////
[route2swingserver]
REGEX=(.)
DEST_KEY=_TCP_ROUTING
FORMAT=logs4customer

/////outputs.conf/////
[indexAndForward]
index = true

[tcpout]
defaultGroup=nothing
maxQueueSize = 10MB

[tcpout:logs4customer]
disabled=false
server=indexerB:9998
compressed = true

[Masa]

Let's keep in mind that this is a kind of hack to make it work in this scenario.

As you can see this will become very custom behavior of Splunk.
Users need to take a responsibility of any upgrade or unexpected issue which might not happen without this change.

1. Disabling regexreplacement at UF could potentially cause other issue for events using regexreplacement. For example, some inputs data other than iis data might count on this feature at UF. So, I do not think this is a good change.

2. Introducing custom route setting might cause after upgrading to newer version which might have new routing rules, and new routing rules might be broken due to this custom route rule. I've seen user experienced unexpected behavior of Splunk and could not tell why after upgrade because of their own custom route in the past. It is rare case. But, it is quite important to keep in mind this. So, whenever doing upgrade, you should check any difference of inputs.conf files contains route attributes in system and builtin app default location.

3. When filing a Support case, it is always good to let Support know this custom settings.

[acharlieh]

I suspect the problem is that the iis sourcetype out of the box uses INDEXED_EXTRACTIONS (which is a form of Structured Parsing) whereas the windows event logs do not.

This diagram I think is mostly accurate: https://wiki.splunk.com/Community:HowIndexingWorks#4._Detail_Diagram_-_UF.2FLWF_to_Indexer

Usually what happens is data is picked up by your Universal Forwarder, and sent to the indexer in fairly raw chunks for parsing. However, when you have something where header of the file determines the field names, like IIS logs and CSV logs, the Universal Forwarder in fact does the structuredParsing (including index time field extraction), and sends fully parsed data to the indexer, this fully parsed data then skips the processors on the indexers that were already performed, notably in this case the regexreplacement processor.

You may be interested in Amrit's perennial How Splunkd Works talk from .conf (there are recordings of 2014 and 2015 floating around), but this is the magic of the other side effect of INDEXED_EXTRACTIONS from a universal forwarder as expressed in this Splunk Reaction.

There are a few different possible solutions you might be able to implement for this issue. As you're routing all of this sourcetype without any filtering to the other indexer, the simplest solution may be to just have the UF send that data straight to that indexer using _TCP_ROUTING in inputs.conf (and of course distributing the other outputs.conf to the UFs).

Another theoretical option is forcing the already parsed data received by the indexer back through the typingQueue instead of skipping straight to the indexQueue (but this quite tricky, I've never done it myself, and you'd probably want some PS on it).