Splunk Enterprise

Forward only specific data from one indexer to another indexer

st4ple
Path Finder

I have a situation where I want to send just the content of one local log file on one indexer ("test_indexer") to another indexer ("production_indexer"). Apart from that, the sending indexer in this scenario ("test_indexer") should continue to function as usual (indexing everything else locally).

My plan was to just add an additional tcpout stanza in outputs.conf (in my case [tcpout:production_indexer] in /opt/splunk/etc/system/local/outputs.conf) and declare the _TCP_ROUTING parameter for the specific stanza in inputs.conf. 

Problem: The sending indexer ("test_indexer") stops indexing any incoming and local data completely after I add the following configurations:

/opt/splunk/etc/system/local/inputs.conf

 

[monitor:///path/to/my/file.log]
index = my_index
sourcetype = my_sourcetype
_TCP_ROUTING = production_indexer

 

/opt/splunk/etc/system/local/outputs.conf

 

[tcpout:production_indexer]
clientCert = $SPLUNK_HOME/etc/auth/server.pem
server = xyz:9998
sslPassword = $abc==
sslVerifyServerCert = false
useSSL = true

 

To me, this behavior is wrong. I am just adding an additional, non-default tcpout stanza (on top of the default one defined in /opt/splunk/etc/system/default/outputs.conf) that is used only by one specific input stanza. According to my understanding, this change should neither impact any other inputs not the default tcpout definition. 

Debugging output before adding the above configuration: 

 

$ splunk btool --debug outputs list
/opt/splunk/etc/system/default/outputs.conf [syslog]
/opt/splunk/etc/system/default/outputs.conf maxEventSize = 1024
/opt/splunk/etc/system/default/outputs.conf priority = <13>
/opt/splunk/etc/system/default/outputs.conf type = udp
/opt/splunk/etc/system/default/outputs.conf [tcpout]
/opt/splunk/etc/system/default/outputs.conf ackTimeoutOnShutdown = 30
/opt/splunk/etc/system/default/outputs.conf autoLBFrequency = 30
/opt/splunk/etc/system/default/outputs.conf autoLBVolume = 0
/opt/splunk/etc/system/default/outputs.conf blockOnCloning = true
/opt/splunk/etc/system/default/outputs.conf blockWarnThreshold = 100
/opt/splunk/etc/system/default/outputs.conf cipherSuite = xyz
/opt/splunk/etc/system/default/outputs.conf compressed = false
/opt/splunk/etc/system/default/outputs.conf connectionTTL = 0
/opt/splunk/etc/system/default/outputs.conf connectionTimeout = 20
/opt/splunk/etc/system/default/outputs.conf disabled = false
/opt/splunk/etc/system/default/outputs.conf dropClonedEventsOnQueueFull = 5
/opt/splunk/etc/system/default/outputs.conf dropEventsOnQueueFull = -1
/opt/splunk/etc/system/default/outputs.conf ecdhCurves = prime256v1, secp384r1, secp521r1
/opt/splunk/etc/system/default/outputs.conf forceTimebasedAutoLB = false
/opt/splunk/etc/system/default/outputs.conf forwardedindex.0.whitelist = .*
/opt/splunk/etc/system/default/outputs.conf forwardedindex.1.blacklist = _.*
/opt/splunk/etc/system/default/outputs.conf forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry|_metrics|_metrics_rollup)
/opt/splunk/etc/system/default/outputs.conf forwardedindex.filter.disable = false
/opt/splunk/etc/system/default/outputs.conf heartbeatFrequency = 30
/opt/splunk/etc/system/default/outputs.conf indexAndForward = false
/opt/splunk/etc/system/default/outputs.conf maxConnectionsPerIndexer = 2
/opt/splunk/etc/system/default/outputs.conf maxFailuresPerInterval = 2
/opt/splunk/etc/system/default/outputs.conf maxQueueSize = auto
/opt/splunk/etc/system/default/outputs.conf readTimeout = 300
/opt/splunk/etc/system/default/outputs.conf secsInFailureInterval = 1
/opt/splunk/etc/system/default/outputs.conf sendCookedData = true
/opt/splunk/etc/system/default/outputs.conf sslQuietShutdown = false
/opt/splunk/etc/system/default/outputs.conf sslVersions = tls1.2
/opt/splunk/etc/system/default/outputs.conf tcpSendBufSz = 0
/opt/splunk/etc/system/default/outputs.conf useACK = false
/opt/splunk/etc/system/default/outputs.conf writeTimeout = 300

 

Debugging output after adding the above configuration: 

 

$ splunk btool --debug outputs list
/opt/splunk/etc/system/default/outputs.conf [syslog]
/opt/splunk/etc/system/default/outputs.conf maxEventSize = 1024
/opt/splunk/etc/system/default/outputs.conf priority = <13>
/opt/splunk/etc/system/default/outputs.conf type = udp
/opt/splunk/etc/system/default/outputs.conf [tcpout]
/opt/splunk/etc/system/default/outputs.conf ackTimeoutOnShutdown = 30
/opt/splunk/etc/system/default/outputs.conf autoLBFrequency = 30
/opt/splunk/etc/system/default/outputs.conf autoLBVolume = 0
/opt/splunk/etc/system/default/outputs.conf blockOnCloning = true
/opt/splunk/etc/system/default/outputs.conf blockWarnThreshold = 100
/opt/splunk/etc/system/default/outputs.conf cipherSuite = xyz
/opt/splunk/etc/system/default/outputs.conf compressed = false
/opt/splunk/etc/system/default/outputs.conf connectionTTL = 0
/opt/splunk/etc/system/default/outputs.conf connectionTimeout = 20
/opt/splunk/etc/system/default/outputs.conf disabled = false
/opt/splunk/etc/system/default/outputs.conf dropClonedEventsOnQueueFull = 5
/opt/splunk/etc/system/default/outputs.conf dropEventsOnQueueFull = -1
/opt/splunk/etc/system/default/outputs.conf ecdhCurves = prime256v1, secp384r1, secp521r1
/opt/splunk/etc/system/default/outputs.conf forceTimebasedAutoLB = false
/opt/splunk/etc/system/default/outputs.conf forwardedindex.0.whitelist = .*
/opt/splunk/etc/system/default/outputs.conf forwardedindex.1.blacklist = _.*
/opt/splunk/etc/system/default/outputs.conf forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry|_metrics|_metrics_rollup)
/opt/splunk/etc/system/default/outputs.conf forwardedindex.filter.disable = false
/opt/splunk/etc/system/default/outputs.conf heartbeatFrequency = 30
/opt/splunk/etc/system/default/outputs.conf indexAndForward = false
/opt/splunk/etc/system/default/outputs.conf maxConnectionsPerIndexer = 2
/opt/splunk/etc/system/default/outputs.conf maxFailuresPerInterval = 2
/opt/splunk/etc/system/default/outputs.conf maxQueueSize = auto
/opt/splunk/etc/system/default/outputs.conf readTimeout = 300
/opt/splunk/etc/system/default/outputs.conf secsInFailureInterval = 1
/opt/splunk/etc/system/default/outputs.conf sendCookedData = true
/opt/splunk/etc/system/default/outputs.conf sslQuietShutdown = false
/opt/splunk/etc/system/default/outputs.conf sslVersions = tls1.2
/opt/splunk/etc/system/default/outputs.conf tcpSendBufSz = 0
/opt/splunk/etc/system/default/outputs.conf useACK = false
/opt/splunk/etc/system/default/outputs.conf writeTimeout = 300
/opt/splunk/etc/system/local/outputs.conf   [tcpout:production_indexer]
/opt/splunk/etc/system/local/outputs.conf   clientCert = $SPLUNK_HOME/etc/auth/server.pem
/opt/splunk/etc/system/local/outputs.conf   server = xyz:9998
/opt/splunk/etc/system/local/outputs.conf   sslPassword = $abc==
/opt/splunk/etc/system/local/outputs.conf   sslVerifyServerCert = false
/opt/splunk/etc/system/local/outputs.conf   useSSL = true

 

Note: Setting  indexAndForward to true is not an option as I really only want to forward the contents of the one specific local log file to the other indexer. 

0 Karma
1 Solution

st4ple
Path Finder

We ended up installing Splunk Universal Forwarder alongside Splunk Enterprise on the same host and just used UF to monitor the one log file whose contents needed to be forwarded to the other Indexer. 

Introduces some overhead but that was not too much of an issue in our case. 

View solution in original post

0 Karma

st4ple
Path Finder

We ended up installing Splunk Universal Forwarder alongside Splunk Enterprise on the same host and just used UF to monitor the one log file whose contents needed to be forwarded to the other Indexer. 

Introduces some overhead but that was not too much of an issue in our case. 

0 Karma

thambisetty
SplunkTrust
SplunkTrust

Give a try below:

indexAndForward = <boolean>
* Set to "true" to index all data locally, in addition to forwarding it.
* This is known as an "index-and-forward" configuration.
* This setting is only available for heavy forwarders.
* This setting is only available at the top level [tcpout] stanza. It
  cannot be overridden in a target group.
* Default: false
[tcpout]
indexAndForward = true

[tcpout:production_indexer]
clientCert = $SPLUNK_HOME/etc/auth/server.pem
server = xyz:9998
sslPassword = $abc==
sslVerifyServerCert = false
useSSL = true

 

————————————
If this helps, give a like below.
0 Karma

st4ple
Path Finder

As I stated in the initial question, setting indexAndForward = true is not really an option since I want to exclusively just forward the content of one log file to another Splunk Indexer. Simply indexing everything on both Indexers is not really satisfactory in this case.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...