Why is my _TCP_ROUTING configuration not working as expected to send log files to two different Splunk instances?

[arjangoos]

Hi,

I want to send my log files to two different Splunk instances, but the data is only only being sent to one of the two environments.

outputs.conf:

[tcpout]
defaultGroup=noredirect
disabled = false
maxQueueSize = 500KB

[tcpout:twd104]
server = 10.9.3.35:9997

[tcpout:splunk_prod]
server = splunk-indexer01.rotterdam.local:9997,splunk-indexer02.rotterdam.local:9997

inputs.conf:

[monitor:///var/log/mule/.../*]
_TCP_ROUTING=splunk_prod
disabled = false
followTail = 0
sourcetype = log4j
index = acc_abkr

[monitor:///var/log/mule/.../*]
_TCP_ROUTING=twd104
disabled = false
followTail = 0
sourcetype = log4j

The data is only send to splunk_prod.

Why is this not working?

[dfronck]

I agree with hgrow. I don't think you can have 2 stanzas dealing with the same log source.

So instead of using _TCP_ROUTING in inputs, use props and transforms.

props.conf (Not sure about the source.)

[source::///var/log/mule/.../*]
TRANSFORMS-routingto2splunks = log_to_splunk1, log_to_splunk2

transforms.conf

[log_to_splunk1]
REGEX = .
DEST_KEY = _TCP_ROUTING
FORMAT = splunk_prod

[log_to_splunk2]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = twd104

[hgrow]

Hi,

My guess is that Splunk ignores your second input because all data is already read with the first input since both inputs are matching the same monitoring pattern.

You don't want two Inputs for the same source. What might help you is a second target group for your input.

What you need is something like:

[tcpout]
defaultGroup= <target_group1> , <target_group2>

[tcpout:<target_group1>]
server= <receiving_server1> , <receiving_server2>

[tcpout:<target_group2>]
server= <receiving_server3>

http://docs.splunk.com/Documentation/Splunk/6.2.4/Forwarding/Configureforwarderswithoutputs.confd

[arjangoos]

this is not working, the data gets sent to one target-group but not to the two groups at the same time.

[hgrow]

Hi arjangoos.

I'm not sure if your problem is stil relevant, but i've done some tests. Here is a working configuration:

inputs.conf

[monitor:///opt/splunkdata/testdata.log]
disabled = false
index = main
sourcetype = test
_TCP_ROUTING = indexer1,indexer2

outputs.conf

[tcpout]
indexAndForward=true
defaultGroup=discard

[tcpout:indexer1]
server=192.168.111.246:9997

[tcpout:indexer2]
server=192.168.111.245:9997

[tcpout:discard]
disabled = true

In addition to my previous posts you define multiple tcpout_groups for the _TCP_ROUTING directly in your input. In this way you can use default routing functionality within the default stanza tcpout for something else.

[hgrow]

That's strange. I wasnt able to rebuild your problem but according to the docs its what you need for data cloning.

Data cloning

To perform data cloning, specify multiple target groups, each in its own stanza. In data cloning, the forwarder sends copies of all its events to the receivers in two or more target groups. Data cloning usually results in similar, but not necessarily exact, copies of data on the receiving indexers. Here's an example of how you set up data cloning:

[tcpout]
defaultGroup=indexer1,indexer2

[tcpout:indexer1]
server=10.1.1.197:9997

[tcpout:indexer2]
server=10.1.1.200:9997

The forwarder will send duplicate data streams to the servers specified in both the indexer1 and indexer2 target groups.

http://docs.splunk.com/Documentation/Splunk/6.2.4/Forwarding/Configureforwarderswithoutputs.confd

Can you post your changed inputs.conf and outputs.conf ?