Forward/Route data from one Splunk Enterprise Instance to other Splunk Enterprise Instance?

[MrWhoztheBoss]

Hi Everyone,

Explaining the installation scenario & requirement first so that the question would make a better sense.

Installation -
Standalone Splunk Enterprise installed on TEST01 server.
Standalone Splunk Enterprise installed on PT01 server.
Task -
Forward/Route data from a specific folder on TEST01 to PT01. All the rest of data should reside on TEST01 only and should be searchable.

This is a business requirement with me. I tried adding [tcpout:PT01] to outputs.conf and _TCP_ROUTING to a [monitor] stanza for that folder on our TEST01 but that ended up sending all the data from TEST01 to PT01 instead of sending just that specific data.
To try a different approach I worked to add transforms, props & outputs .conf files according to this doc - Route and filter data but that didn't helped and apparently induced some instability on TEST01 Splunk Enterprise Installation as it was not able to stop and start correctly.

Any guidance on how I can achieve this would be very much helpful ❤️

[richgalloway]

It sounds like Route and Filter  (https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad) is what you want.    How did it not help?  What "instability" did it create?  Can you share the _TCP_ROUTING config that didn't work?  Perhaps we can fix it for you.

[MrWhoztheBoss]

Hi @richgalloway ,

Thanks for extending a hand of help on this.

Earlier I had added only [tcpout:PT01] stanza with server details in outputs.conf which ended up sending all the data to PT01 and nothing was indexed locally on TEST01 (Expected).
To counter this I went ahead and tried configs like this Filter and route event data to target groups editing props, transforms & outputs which added some absurd behavior, like when trying to restart it was timing out trying to stop and I had to start manually.

On exploring more about routing I found Perform selective indexing and forwarding was the exact kind of behavior I was looking up to implement. So I edited the inputs & outputs files on TEST01 as below.

---------- inputs.conf ----------

[default]
host=test01

[batch:///opt/splunk/data/Test]
index=main
sourcetype=testing-kvp
move_policy=sinkhole
_INDEX_AND_FORWARD_ROUTING=local

[batch:///opt/splunk/data/Test2PT]
index=main
sourcetype=testing2pt-kvp
move_policy=sinkhole
_TCP_ROUTING=pt01

---------- outputs.conf ----------

[tcpout]
defaultGroup=noforward
disabled=false

[indexAndForward]
index=true
selectiveIndexing=true

[tcpout:pt01]
server=PT01:9997

This move solved my requirement, that is indexed data locally on TEST01 and forward some data to PT01 as well but raised a new set of problems.
As mentioned our TEST01 is a standalone Splunk Enterprise installation, it has data being forwarded to it from UF on other hosts, scripted inputs, rest endpoints, etc. data inputs are configured which gets indexed daily over it. After adding this configuration all other data inputs apart from the one's mentioned in inputs.conf stopped indexing totally.

Can I get some help to reach the end as on how I can index literally all the data on TEST01 locally and just monitor forward files from one folder, in this case "/opt/splunk/data/Test2PT" to PT01 ?