Hi there,
We're trying to have a splunk forwarder to send data to an intermediate splunk heavy forwarder that
clones data to different indexer cluster groups based on the sourcetype.
sourcetype1 send data to indexercluster1 (default routing)
sourcetype2 send data to indexercluster1 and indexercluster2
We have set the following below, but data is only indexed in indexercluster1 for some reason.
Never sent to indexercluster2 for sourcetype2 as wanted.
Any idea why ?
Thanks a lot for any help
props.conf
[sourcetype2]
TRANSFORMS-routing=TR_routing_sourcetype2
transforms.conf
[TR_routing_sourcetype2]
REGEX=.
DEST_KEY=_TCP_ROUTING
FORMAT=indexercluster1,indexercluster2
outputs.conf
[tcpout]
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist =
forwardedindex.2.whitelist =
forwardedindex.filter.disable = false
defaultGroup = indexercluster1
indexAndForward = false
[tcpout:indexercluster1]
disabled = false
maxQueueSize = 6MB
server = 10.0.1.1:9997,10.0.1.2:9997,10.0.1.3:9997
sslCertPath = $SPLUNK_HOME/etc/auth/server.pem
sslPassword = somepassword
sslRootCAPath = $SPLUNK_HOME/etc/auth/cacert.pem
sslVerifyServerCert = false
useACK = true
[tcpout:indexercluster2]
disabled = false
maxQueueSize = 6MB
server = 10.0.2.1:9997,10.0.2.2:9997,10.0.2.3:9997
sslCertPath = $SPLUNK_HOME/etc/auth/server.pem
sslPassword = somepassword
sslRootCAPath = $SPLUNK_HOME/etc/auth/cacert.pem
sslVerifyServerCert = false
useACK = true
inputs.conf
[default]
#_TCP_ROUTING = *
[SSL]
rootCA = $SPLUNK_HOME/etc/auth/cacert.pem
serverCert = $SPLUNK_HOME/etc/auth/server.pem
password = somepassword
[splunktcp-ssl:9997]
Hi,
which version are you using?
In 6.4.3, according to inputs.conf you can use multiple groups... haven't tested.
Don't know whether this works in props/transforms.
One other option I have in mind is to use CLONE_SOURCETYPE and route this new sourcetype to an additional group.
It's possible to do this
props.conf:
[orig_sourcetype]
TRANSFORMS-clone = clone_orig
[cloned_sourcetype]
TRANSFORMS-route = route_clone
transforms.conf
[clone_orig]
REGEX = (.*)
FORMAT = $1
DEST_KEY = _raw
CLONE_SOURCETYPE = cloned_sourcetype
[route_clone]
REGEX = .
DEST_KEY=_TCP_ROUTING
FORMAT=indexercluster2
outputs.conf should stay the same...? Does it work for you?
You should be able to add more transforms for the "new" sourcetype... maybe even renaming it again?
Or use a sourcetype alias on the second indexers group.
HTH,
Holger
Have you tried breaking it into two individual transforms? e.g.
props.conf
[sourcetype2]
TRANSFORMS-routing=TR_routing_sourcetype2IDXC1,TR_routing_sourcetype2IDXC2
transforms.conf:
[TR_routing_sourcetype2IDXC1]
REGEX=.
DEST_KEY=_TCP_ROUTING
FORMAT=indexercluster1
[TR_routing_sourcetype2IDXC2]
REGEX=.
DEST_KEY=_TCP_ROUTING
FORMAT=indexercluster2
Thanks dgrubb, much appreciated. We tried this, and we still have data forwarded only to the defaultGroup indexercluster1. No sourcetype2 in indexercluster2.
It was working before when inputs.conf had the _TCP_ROUTING = *, but then all data was sent to all outputs.conf groups whatever the sourcetype, which was not the behaviour we wanted either.
We would like to send all data by default to indexercluster1, and for a selected number of sourcetypes to go to indexercluster1 and indexercluster2.
Any help would be appreciated.