how to forward secure log on the peer node to another splunk enterprise

[bestSplunker]

hello everyone, forgive my English

i have a splunk indexer cluster (3 peer + master node + 1 search header), now ,I don't want to forward the secure log on the peer node to the indexer cluster, I want to forward the secure log on the peer node to another splunk enterprise(alone instance). I tried the following method, please point out my mistake:

1、Point all peer nodes to the deployment server and use the deployment server to distribute the apps.

2、use deployment server put following apss to all peer node:

path on the DS : /opt/splunk/etc/deployment-apps/linux/local/inputs.conf

[monitor:///var/log/secure]
index = linux
sourcetype = linux_secure

path on the DS : /opt/splunk/etc/deployment-apps/linux/local/outputs.conf

[tcpout:test1]
server = 10.10.20.100:9997

3、push apps to all peer node by deployment-server

Something unexpected happened:

All logs originally forwarded to the indexer cluster changed the forwarding route ,The peer node forwards them all to a alone splunk instance.(10.10.20.100).

I don't know why this happened, I think that logs from other hosts should be forwarded to the indexer cluster in addition to the secure logs on the peer nodes.But this is not the case, the logs arriving at the peer node are all routed to the alone splunk instance.This means that the wrong configuration results in a change in peer routing

Does anyone know how to solve this problem? All help would be greatly appreciated

[jnudell_2]

Hi @bestSplunker ,
You will need to use _TCP_ROUTING to be able to do this. You can read the documentation on how to use this feature (https://docs.splunk.com/Documentation/Splunk/7.3.0/Forwarding/Routeandfilterdatad#Route_inputs_to_sp...) and then configure your inputs.conf and outputs.conf accordingly

In your case, it would look something like this
inputs.conf

[monitor:///var/log/secure]

index = linux

sourcetype = linux_secure

_TCP_ROUTING = secure

outputs.conf

[tcpout]

defaultGroup = default

[tcpout:secure]

server = 10.10.20.100:9997

[tcpout:default]

server = <cluster member ip 1>:9997, <cluster member ip 2>:9997, <cluster member ip 3>:9997

[bestSplunker]

By default, there is no tcpout on all peers. If I need to forward the secure log of the peer to another splunk instance, I need to add the tcpout of the indexer cluster to outputs.conf and put it Set as the default group, right?

[jnudell_2]

Additionally, if you're talking about indexer cluster members as "peers", then they should NOT be managed by the deployment server. That's what the cluster master is for.

[jnudell_2]

If by peers, you mean indexer cluster members, then no. What host is the secure log located on? If it's on your clustered indexers, and you want to send the logs to a standalone server, you would only configure the [tcpout:secure] stanza from the answer, and not the [tcpout] and [tcpout:default] stanzas. You would still need the inputs.conf stanza as well.

[bestSplunker]

What host is the secure log located on.
secure log on the peer node.
When I followed your approach, one of the peer node had the following error message:

peer node : 172.25.105.159

connect to 172.25.105.159:9997 failed

Forwarding to indexer group default blocked for 370 secounds

I suspect this error occurs because they forwarded the data to their own port.

[jnudell_2]

I would revert back to the original settings. 172.25.105.159 is one of the indexer cluster members? That should not be configured in outputs.conf for any of the indexers anyway. I thought you were trying to send to 10.10.20.100?

[bestSplunker]

@jnudell_2 I'm sorry, forgive my English. I didn't express it clearly enough. i have a indexer cluster that contain 3 peer nodes (peer ip: 172.25.105.158/159/160)。the linux secure log on the 3 peer nodes（/var/log/secure）,by the default, The secure log will be forwarded to the indexer cluster if i only configured inputs.conf. and I can search for them using the search header.
now. i want to forwarder them to another standalone instance(10.10.20.100). So how do I forward the secure logs of these 3 peers to a standalone splunk instance.