Forward only specific data from one indexer to another indexer

[st4ple]

I have a situation where I want to send just the content of one local log file on one indexer ("test_indexer") to another indexer ("production_indexer"). Apart from that, the sending indexer in this scenario ("test_indexer") should continue to function as usual (indexing everything else locally).

My plan was to just add an additional tcpout stanza in outputs.conf (in my case [tcpout:production_indexer] in /opt/splunk/etc/system/local/outputs.conf) and declare the _TCP_ROUTING parameter for the specific stanza in inputs.conf. 

Problem: The sending indexer ("test_indexer") stops indexing any incoming and local data completely after I add the following configurations:

/opt/splunk/etc/system/local/inputs.conf

[monitor:///path/to/my/file.log]
index = my_index
sourcetype = my_sourcetype
_TCP_ROUTING = production_indexer

/opt/splunk/etc/system/local/outputs.conf

[tcpout:production_indexer]
clientCert = $SPLUNK_HOME/etc/auth/server.pem
server = xyz:9998
sslPassword = $abc==
sslVerifyServerCert = false
useSSL = true

To me, this behavior is wrong. I am just adding an additional, non-default tcpout stanza (on top of the default one defined in /opt/splunk/etc/system/default/outputs.conf) that is used only by one specific input stanza. According to my understanding, this change should neither impact any other inputs not the default tcpout definition. 

Debugging output before adding the above configuration: 

$ splunk btool --debug outputs list
/opt/splunk/etc/system/default/outputs.conf [syslog]
/opt/splunk/etc/system/default/outputs.conf maxEventSize = 1024
/opt/splunk/etc/system/default/outputs.conf priority = <13>
/opt/splunk/etc/system/default/outputs.conf type = udp
/opt/splunk/etc/system/default/outputs.conf [tcpout]
/opt/splunk/etc/system/default/outputs.conf ackTimeoutOnShutdown = 30
/opt/splunk/etc/system/default/outputs.conf autoLBFrequency = 30
/opt/splunk/etc/system/default/outputs.conf autoLBVolume = 0
/opt/splunk/etc/system/default/outputs.conf blockOnCloning = true
/opt/splunk/etc/system/default/outputs.conf blockWarnThreshold = 100
/opt/splunk/etc/system/default/outputs.conf cipherSuite = xyz
/opt/splunk/etc/system/default/outputs.conf compressed = false
/opt/splunk/etc/system/default/outputs.conf connectionTTL = 0
/opt/splunk/etc/system/default/outputs.conf connectionTimeout = 20
/opt/splunk/etc/system/default/outputs.conf disabled = false
/opt/splunk/etc/system/default/outputs.conf dropClonedEventsOnQueueFull = 5
/opt/splunk/etc/system/default/outputs.conf dropEventsOnQueueFull = -1
/opt/splunk/etc/system/default/outputs.conf ecdhCurves = prime256v1, secp384r1, secp521r1
/opt/splunk/etc/system/default/outputs.conf forceTimebasedAutoLB = false
/opt/splunk/etc/system/default/outputs.conf forwardedindex.0.whitelist = .*
/opt/splunk/etc/system/default/outputs.conf forwardedindex.1.blacklist = _.*
/opt/splunk/etc/system/default/outputs.conf forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry|_metrics|_metrics_rollup)
/opt/splunk/etc/system/default/outputs.conf forwardedindex.filter.disable = false
/opt/splunk/etc/system/default/outputs.conf heartbeatFrequency = 30
/opt/splunk/etc/system/default/outputs.conf indexAndForward = false
/opt/splunk/etc/system/default/outputs.conf maxConnectionsPerIndexer = 2
/opt/splunk/etc/system/default/outputs.conf maxFailuresPerInterval = 2
/opt/splunk/etc/system/default/outputs.conf maxQueueSize = auto
/opt/splunk/etc/system/default/outputs.conf readTimeout = 300
/opt/splunk/etc/system/default/outputs.conf secsInFailureInterval = 1
/opt/splunk/etc/system/default/outputs.conf sendCookedData = true
/opt/splunk/etc/system/default/outputs.conf sslQuietShutdown = false
/opt/splunk/etc/system/default/outputs.conf sslVersions = tls1.2
/opt/splunk/etc/system/default/outputs.conf tcpSendBufSz = 0
/opt/splunk/etc/system/default/outputs.conf useACK = false
/opt/splunk/etc/system/default/outputs.conf writeTimeout = 300

Debugging output after adding the above configuration: 

$ splunk btool --debug outputs list
/opt/splunk/etc/system/default/outputs.conf [syslog]
/opt/splunk/etc/system/default/outputs.conf maxEventSize = 1024
/opt/splunk/etc/system/default/outputs.conf priority = <13>
/opt/splunk/etc/system/default/outputs.conf type = udp
/opt/splunk/etc/system/default/outputs.conf [tcpout]
/opt/splunk/etc/system/default/outputs.conf ackTimeoutOnShutdown = 30
/opt/splunk/etc/system/default/outputs.conf autoLBFrequency = 30
/opt/splunk/etc/system/default/outputs.conf autoLBVolume = 0
/opt/splunk/etc/system/default/outputs.conf blockOnCloning = true
/opt/splunk/etc/system/default/outputs.conf blockWarnThreshold = 100
/opt/splunk/etc/system/default/outputs.conf cipherSuite = xyz
/opt/splunk/etc/system/default/outputs.conf compressed = false
/opt/splunk/etc/system/default/outputs.conf connectionTTL = 0
/opt/splunk/etc/system/default/outputs.conf connectionTimeout = 20
/opt/splunk/etc/system/default/outputs.conf disabled = false
/opt/splunk/etc/system/default/outputs.conf dropClonedEventsOnQueueFull = 5
/opt/splunk/etc/system/default/outputs.conf dropEventsOnQueueFull = -1
/opt/splunk/etc/system/default/outputs.conf ecdhCurves = prime256v1, secp384r1, secp521r1
/opt/splunk/etc/system/default/outputs.conf forceTimebasedAutoLB = false
/opt/splunk/etc/system/default/outputs.conf forwardedindex.0.whitelist = .*
/opt/splunk/etc/system/default/outputs.conf forwardedindex.1.blacklist = _.*
/opt/splunk/etc/system/default/outputs.conf forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry|_metrics|_metrics_rollup)
/opt/splunk/etc/system/default/outputs.conf forwardedindex.filter.disable = false
/opt/splunk/etc/system/default/outputs.conf heartbeatFrequency = 30
/opt/splunk/etc/system/default/outputs.conf indexAndForward = false
/opt/splunk/etc/system/default/outputs.conf maxConnectionsPerIndexer = 2
/opt/splunk/etc/system/default/outputs.conf maxFailuresPerInterval = 2
/opt/splunk/etc/system/default/outputs.conf maxQueueSize = auto
/opt/splunk/etc/system/default/outputs.conf readTimeout = 300
/opt/splunk/etc/system/default/outputs.conf secsInFailureInterval = 1
/opt/splunk/etc/system/default/outputs.conf sendCookedData = true
/opt/splunk/etc/system/default/outputs.conf sslQuietShutdown = false
/opt/splunk/etc/system/default/outputs.conf sslVersions = tls1.2
/opt/splunk/etc/system/default/outputs.conf tcpSendBufSz = 0
/opt/splunk/etc/system/default/outputs.conf useACK = false
/opt/splunk/etc/system/default/outputs.conf writeTimeout = 300
/opt/splunk/etc/system/local/outputs.conf   [tcpout:production_indexer]
/opt/splunk/etc/system/local/outputs.conf   clientCert = $SPLUNK_HOME/etc/auth/server.pem
/opt/splunk/etc/system/local/outputs.conf   server = xyz:9998
/opt/splunk/etc/system/local/outputs.conf   sslPassword = $abc==
/opt/splunk/etc/system/local/outputs.conf   sslVerifyServerCert = false
/opt/splunk/etc/system/local/outputs.conf   useSSL = true

Note: Setting  indexAndForward to true is not an option as I really only want to forward the contents of the one specific local log file to the other indexer. 

[st4ple · Accepted solution]

We ended up installing Splunk Universal Forwarder alongside Splunk Enterprise on the same host and just used UF to monitor the one log file whose contents needed to be forwarded to the other Indexer. 

Introduces some overhead but that was not too much of an issue in our case. 

[thambisetty]

Give a try below:

indexAndForward = <boolean>
* Set to "true" to index all data locally, in addition to forwarding it.
* This is known as an "index-and-forward" configuration.
* This setting is only available for heavy forwarders.
* This setting is only available at the top level [tcpout] stanza. It
  cannot be overridden in a target group.
* Default: false

[tcpout]
indexAndForward = true

[tcpout:production_indexer]
clientCert = $SPLUNK_HOME/etc/auth/server.pem
server = xyz:9998
sslPassword = $abc==
sslVerifyServerCert = false
useSSL = true

[st4ple]

As I stated in the initial question, setting indexAndForward = true is not really an option since I want to exclusively just forward the content of one log file to another Splunk Indexer. Simply indexing everything on both Indexers is not really satisfactory in this case.