Cloning set of data to specified Splunk indexer

andrey2007 — Mon, 28 Sep 2020 19:18:07 GMT

Hello, all
I faced with problem of tcp routing. As I know there is a bug with _TCP_ROUTING of modular inputs.

How can I make Splunk forward data from my heavy forwarder such way that all data routes to first group of indexers but some data set also clones to another server. On my HF I have a lot of apps with data sources and its specification.

I tried to specify only blacklist in outputs.conf for clone group but it doesnot work.
so now my configs looks like
what is way to resolve this case? -

inputs.conf (in etc\apps\myapp\local) for data which should be cloned

[script://$SPLUNK_HOME/bin/scripts/script.sh]
disabled = false
index=myindex
sourcetype=routedsourcetype

props.conf in etc\system\local

[default]
TRANSFORMS-routing= allRouting

[routedsourcetype]
TRANSFORMS-routing= specrouting

transforms.conf in etc\system\local

[allRouting]
REGEX=(.)
DEST_KEY=_TCP_ROUTING
FORMAT=splunk_idx

[specrouting]
REGEX=(.)
DEST_KEY=_TCP_ROUTING
FORMAT=splunk_idx,routing_idx

outputs.conf in \etc\apps\outputapp\local

[tcpout]
defaultGroup = splunk_idx
maxQueueSize = auto
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = (_audit|_internal)
forwardedindex.filter.disable = false
indexAndForward = false
autoLBFrequency = 30
blockOnCloning = true
compressed = false
disabled = false
dropClonedEventsOnQueueFull = 5
dropEventsOnQueueFull = -1
heartbeatFrequency = 90
maxFailuresPerInterval = 2
secsInFailureInterval = 1
maxConnectionsPerIndexer = 2
forceTimebasedAutoLB = false
sendCookedData = true
connectionTimeout = 20 
readTimeout = 300
writeTimeout = 300 
useACK = true

[tcpout:splunk_idx]
server=splunk-idx01:9997,splunk-idx02:9997
useACK=true
autoLB=true

[tcpout:routing_idx]
disabled=false
useACK=false
server=routing_server:9997

Re: Cloning set of data to specified Splunk indexer

Jason — Mon, 30 Mar 2015 11:31:28 GMT

That approach looks correct to me. Define a default output group ( splunk_idx in your config), create another output group ( routing_idx ), and selectively send some data to it via a TRANSFORMS modifying _TCP_ROUTING. This statement is redundant and can be removed:

[default]
TRANSFORMS-routing= allRouting

I would ensure that you are putting these settings on the Splunk instance that is doing the Parsing phase of the data lifecycle. (The first Heavy Forwarder or Indexer that the data hits.) Then, check that the relevant settings are actually going into place, by using btool. Etc/system/local is the location that can not be overridden by any other location, except if you are on a cluster peer. See the precedence rules.

topic Re: Cloning set of data to specified Splunk indexer in Getting Data In

Cloning set of data to specified Splunk indexer

Re: Cloning set of data to specified Splunk indexer