Splunk Enterprise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Ask a Question
Solved! Jump to solution

Forward only specific data from one indexer to another indexer

st4ple
Path Finder
‎09-15-2020 12:22 AM

I have a situation where I want to send just the content of one local log file on one indexer ("test_indexer") to another indexer ("production_indexer"). Apart from that, the sending indexer in this scenario ("test_indexer") should continue to function as usual (indexing everything else locally).

My plan was to just add an additional tcpout stanza in outputs.conf (in my case [tcpout:production_indexer] in /opt/splunk/etc/system/local/outputs.conf) and declare the _TCP_ROUTING parameter for the specific stanza in inputs.conf. 

Problem: The sending indexer ("test_indexer") stops indexing any incoming and local data completely after I add the following configurations:

/opt/splunk/etc/system/local/inputs.conf

 

[monitor:///path/to/my/file.log]
index = my_index
sourcetype = my_sourcetype
_TCP_ROUTING = production_indexer

 

/opt/splunk/etc/system/local/outputs.conf

 

[tcpout:production_indexer]
clientCert = $SPLUNK_HOME/etc/auth/server.pem
server = xyz:9998
sslPassword = $abc==
sslVerifyServerCert = false
useSSL = true

 

To me, this behavior is wrong. I am just adding an additional, non-default tcpout stanza (on top of the default one defined in /opt/splunk/etc/system/default/outputs.conf) that is used only by one specific input stanza. According to my understanding, this change should neither impact any other inputs not the default tcpout definition. 

Debugging output before adding the above configuration: 

 

$ splunk btool --debug outputs list
/opt/splunk/etc/system/default/outputs.conf [syslog]
/opt/splunk/etc/system/default/outputs.conf maxEventSize = 1024
/opt/splunk/etc/system/default/outputs.conf priority = <13>
/opt/splunk/etc/system/default/outputs.conf type = udp
/opt/splunk/etc/system/default/outputs.conf [tcpout]
/opt/splunk/etc/system/default/outputs.conf ackTimeoutOnShutdown = 30
/opt/splunk/etc/system/default/outputs.conf autoLBFrequency = 30
/opt/splunk/etc/system/default/outputs.conf autoLBVolume = 0
/opt/splunk/etc/system/default/outputs.conf blockOnCloning = true
/opt/splunk/etc/system/default/outputs.conf blockWarnThreshold = 100
/opt/splunk/etc/system/default/outputs.conf cipherSuite = xyz
/opt/splunk/etc/system/default/outputs.conf compressed = false
/opt/splunk/etc/system/default/outputs.conf connectionTTL = 0
/opt/splunk/etc/system/default/outputs.conf connectionTimeout = 20
/opt/splunk/etc/system/default/outputs.conf disabled = false
/opt/splunk/etc/system/default/outputs.conf dropClonedEventsOnQueueFull = 5
/opt/splunk/etc/system/default/outputs.conf dropEventsOnQueueFull = -1
/opt/splunk/etc/system/default/outputs.conf ecdhCurves = prime256v1, secp384r1, secp521r1
/opt/splunk/etc/system/default/outputs.conf forceTimebasedAutoLB = false
/opt/splunk/etc/system/default/outputs.conf forwardedindex.0.whitelist = .*
/opt/splunk/etc/system/default/outputs.conf forwardedindex.1.blacklist = _.*
/opt/splunk/etc/system/default/outputs.conf forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry|_metrics|_metrics_rollup)
/opt/splunk/etc/system/default/outputs.conf forwardedindex.filter.disable = false
/opt/splunk/etc/system/default/outputs.conf heartbeatFrequency = 30
/opt/splunk/etc/system/default/outputs.conf indexAndForward = false
/opt/splunk/etc/system/default/outputs.conf maxConnectionsPerIndexer = 2
/opt/splunk/etc/system/default/outputs.conf maxFailuresPerInterval = 2
/opt/splunk/etc/system/default/outputs.conf maxQueueSize = auto
/opt/splunk/etc/system/default/outputs.conf readTimeout = 300
/opt/splunk/etc/system/default/outputs.conf secsInFailureInterval = 1
/opt/splunk/etc/system/default/outputs.conf sendCookedData = true
/opt/splunk/etc/system/default/outputs.conf sslQuietShutdown = false
/opt/splunk/etc/system/default/outputs.conf sslVersions = tls1.2
/opt/splunk/etc/system/default/outputs.conf tcpSendBufSz = 0
/opt/splunk/etc/system/default/outputs.conf useACK = false
/opt/splunk/etc/system/default/outputs.conf writeTimeout = 300

 

Debugging output after adding the above configuration: 

 

$ splunk btool --debug outputs list
/opt/splunk/etc/system/default/outputs.conf [syslog]
/opt/splunk/etc/system/default/outputs.conf maxEventSize = 1024
/opt/splunk/etc/system/default/outputs.conf priority = <13>
/opt/splunk/etc/system/default/outputs.conf type = udp
/opt/splunk/etc/system/default/outputs.conf [tcpout]
/opt/splunk/etc/system/default/outputs.conf ackTimeoutOnShutdown = 30
/opt/splunk/etc/system/default/outputs.conf autoLBFrequency = 30
/opt/splunk/etc/system/default/outputs.conf autoLBVolume = 0
/opt/splunk/etc/system/default/outputs.conf blockOnCloning = true
/opt/splunk/etc/system/default/outputs.conf blockWarnThreshold = 100
/opt/splunk/etc/system/default/outputs.conf cipherSuite = xyz
/opt/splunk/etc/system/default/outputs.conf compressed = false
/opt/splunk/etc/system/default/outputs.conf connectionTTL = 0
/opt/splunk/etc/system/default/outputs.conf connectionTimeout = 20
/opt/splunk/etc/system/default/outputs.conf disabled = false
/opt/splunk/etc/system/default/outputs.conf dropClonedEventsOnQueueFull = 5
/opt/splunk/etc/system/default/outputs.conf dropEventsOnQueueFull = -1
/opt/splunk/etc/system/default/outputs.conf ecdhCurves = prime256v1, secp384r1, secp521r1
/opt/splunk/etc/system/default/outputs.conf forceTimebasedAutoLB = false
/opt/splunk/etc/system/default/outputs.conf forwardedindex.0.whitelist = .*
/opt/splunk/etc/system/default/outputs.conf forwardedindex.1.blacklist = _.*
/opt/splunk/etc/system/default/outputs.conf forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry|_metrics|_metrics_rollup)
/opt/splunk/etc/system/default/outputs.conf forwardedindex.filter.disable = false
/opt/splunk/etc/system/default/outputs.conf heartbeatFrequency = 30
/opt/splunk/etc/system/default/outputs.conf indexAndForward = false
/opt/splunk/etc/system/default/outputs.conf maxConnectionsPerIndexer = 2
/opt/splunk/etc/system/default/outputs.conf maxFailuresPerInterval = 2
/opt/splunk/etc/system/default/outputs.conf maxQueueSize = auto
/opt/splunk/etc/system/default/outputs.conf readTimeout = 300
/opt/splunk/etc/system/default/outputs.conf secsInFailureInterval = 1
/opt/splunk/etc/system/default/outputs.conf sendCookedData = true
/opt/splunk/etc/system/default/outputs.conf sslQuietShutdown = false
/opt/splunk/etc/system/default/outputs.conf sslVersions = tls1.2
/opt/splunk/etc/system/default/outputs.conf tcpSendBufSz = 0
/opt/splunk/etc/system/default/outputs.conf useACK = false
/opt/splunk/etc/system/default/outputs.conf writeTimeout = 300
/opt/splunk/etc/system/local/outputs.conf   [tcpout:production_indexer]
/opt/splunk/etc/system/local/outputs.conf   clientCert = $SPLUNK_HOME/etc/auth/server.pem
/opt/splunk/etc/system/local/outputs.conf   server = xyz:9998
/opt/splunk/etc/system/local/outputs.conf   sslPassword = $abc==
/opt/splunk/etc/system/local/outputs.conf   sslVerifyServerCert = false
/opt/splunk/etc/system/local/outputs.conf   useSSL = true

 

Note: Setting  indexAndForward to true is not an option as I really only want to forward the contents of the one specific local log file to the other indexer. 

0 Karma
Reply
1 Solution
Solved! Jump to solution
Solution

st4ple
Path Finder
‎11-02-2020 12:08 AM

We ended up installing Splunk Universal Forwarder alongside Splunk Enterprise on the same host and just used UF to monitor the one log file whose contents needed to be forwarded to the other Indexer. 

Introduces some overhead but that was not too much of an issue in our case. 

View solution in original post

0 Karma
Reply
Solved! Jump to solution
Solution

st4ple
Path Finder
‎11-02-2020 12:08 AM

We ended up installing Splunk Universal Forwarder alongside Splunk Enterprise on the same host and just used UF to monitor the one log file whose contents needed to be forwarded to the other Indexer. 

Introduces some overhead but that was not too much of an issue in our case. 

0 Karma
Reply
Solved! Jump to solution

thambisetty
SplunkTrust
SplunkTrust
‎09-15-2020 12:33 AM

Give a try below:

indexAndForward = <boolean>
* Set to "true" to index all data locally, in addition to forwarding it.
* This is known as an "index-and-forward" configuration.
* This setting is only available for heavy forwarders.
* This setting is only available at the top level [tcpout] stanza. It
  cannot be overridden in a target group.
* Default: false
[tcpout]
indexAndForward = true

[tcpout:production_indexer]
clientCert = $SPLUNK_HOME/etc/auth/server.pem
server = xyz:9998
sslPassword = $abc==
sslVerifyServerCert = false
useSSL = true

 

————————————
If this helps, give a like below.
0 Karma
Reply
Solved! Jump to solution

st4ple
Path Finder
‎11-02-2020 12:05 AM

As I stated in the initial question, setting indexAndForward = true is not really an option since I want to exclusively just forward the content of one log file to another Splunk Indexer. Simply indexing everything on both Indexers is not really satisfactory in this case.

0 Karma
Reply
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In December, the Splunk Threat Research Team had 1 release of new security content via the Enterprise Security ...

Why am I not seeing the finding in Splunk Enterprise Security Analyst Queue?

(This is the first of a series of 2 blogs). Splunk Enterprise Security is a fantastic tool that offers robust ...

Index This | What are the 12 Days of Splunk-mas?

December 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...