Getting Data In

Cloning set of data to specified Splunk indexer

andrey2007
Contributor

Hello, all
I faced with problem of tcp routing. As I know there is a bug with _TCP_ROUTING of modular inputs.

How can I make Splunk forward data from my heavy forwarder such way that all data routes to first group of indexers but some data set also clones to another server. On my HF I have a lot of apps with data sources and its specification.

I tried to specify only blacklist in outputs.conf for clone group but it doesnot work.
so now my configs looks like
what is way to resolve this case? -

inputs.conf (in etc\apps\myapp\local) for data which should be cloned

[script://$SPLUNK_HOME/bin/scripts/script.sh]
disabled = false
index=myindex
sourcetype=routedsourcetype

props.conf in etc\system\local

[default]
TRANSFORMS-routing= allRouting

[routedsourcetype]
TRANSFORMS-routing= specrouting

transforms.conf in etc\system\local

[allRouting]
REGEX=(.)
DEST_KEY=_TCP_ROUTING
FORMAT=splunk_idx

[specrouting]
REGEX=(.)
DEST_KEY=_TCP_ROUTING
FORMAT=splunk_idx,routing_idx

outputs.conf in \etc\apps\outputapp\local

[tcpout]
defaultGroup = splunk_idx
maxQueueSize = auto
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = (_audit|_internal)
forwardedindex.filter.disable = false
indexAndForward = false
autoLBFrequency = 30
blockOnCloning = true
compressed = false
disabled = false
dropClonedEventsOnQueueFull = 5
dropEventsOnQueueFull = -1
heartbeatFrequency = 90
maxFailuresPerInterval = 2
secsInFailureInterval = 1
maxConnectionsPerIndexer = 2
forceTimebasedAutoLB = false
sendCookedData = true
connectionTimeout = 20 
readTimeout = 300
writeTimeout = 300 
useACK = true

[tcpout:splunk_idx]
server=splunk-idx01:9997,splunk-idx02:9997
useACK=true
autoLB=true

[tcpout:routing_idx]
disabled=false
useACK=false
server=routing_server:9997
0 Karma
1 Solution

Jason
Motivator

That approach looks correct to me. Define a default output group ( splunk_idx in your config), create another output group ( routing_idx ), and selectively send some data to it via a TRANSFORMS modifying _TCP_ROUTING. This statement is redundant and can be removed:

[default]
TRANSFORMS-routing= allRouting

I would ensure that you are putting these settings on the Splunk instance that is doing the Parsing phase of the data lifecycle. (The first Heavy Forwarder or Indexer that the data hits.) Then, check that the relevant settings are actually going into place, by using btool. Etc/system/local is the location that can not be overridden by any other location, except if you are on a cluster peer. See the precedence rules.

View solution in original post

Jason
Motivator

That approach looks correct to me. Define a default output group ( splunk_idx in your config), create another output group ( routing_idx ), and selectively send some data to it via a TRANSFORMS modifying _TCP_ROUTING. This statement is redundant and can be removed:

[default]
TRANSFORMS-routing= allRouting

I would ensure that you are putting these settings on the Splunk instance that is doing the Parsing phase of the data lifecycle. (The first Heavy Forwarder or Indexer that the data hits.) Then, check that the relevant settings are actually going into place, by using btool. Etc/system/local is the location that can not be overridden by any other location, except if you are on a cluster peer. See the precedence rules.

Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...