Splunk Enterprise

Forward/Route data from one Splunk Enterprise Instance to other Splunk Enterprise Instance?

MrWhoztheBoss
Explorer

Hi Everyone,

Explaining the installation scenario & requirement first so that the question would make a better sense.

Installation -
Standalone Splunk Enterprise installed on TEST01 server.
Standalone Splunk Enterprise installed on PT01 server.
Task -
Forward/Route data from a specific folder on TEST01 to PT01. All the rest of data should reside on TEST01 only and should be searchable.

This is a business requirement with me. I tried adding [tcpout:PT01] to outputs.conf and _TCP_ROUTING to a [monitor] stanza for that folder on our TEST01 but that ended up sending all the data from TEST01 to PT01 instead of sending just that specific data.
To try a different approach I worked to add transforms, props & outputs .conf files according to this doc - Route and filter data but that didn't helped and apparently induced some instability on TEST01 Splunk Enterprise Installation as it was not able to stop and start correctly.

Any guidance on how I can achieve this would be very much helpful ❤️

Labels (2)
Tags (2)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

It sounds like Route and Filter  (https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad) is what you want.    How did it not help?  What "instability" did it create?  Can you share the _TCP_ROUTING config that didn't work?  Perhaps we can fix it for you.

---
If this reply helps you, Karma would be appreciated.
0 Karma

MrWhoztheBoss
Explorer

Hi @richgalloway ,

Thanks for extending a hand of help on this.

Earlier I had added only [tcpout:PT01] stanza with server details in outputs.conf which ended up sending all the data to PT01 and nothing was indexed locally on TEST01 (Expected).
To counter this I went ahead and tried configs like this Filter and route event data to target groups editing props, transforms & outputs which added some absurd behavior, like when trying to restart it was timing out trying to stop and I had to start manually.

On exploring more about routing I found Perform selective indexing and forwarding was the exact kind of behavior I was looking up to implement. So I edited the inputs & outputs files on TEST01 as below.

---------- inputs.conf ----------

[default]
host=test01

[batch:///opt/splunk/data/Test]
index=main
sourcetype=testing-kvp
move_policy=sinkhole
_INDEX_AND_FORWARD_ROUTING=local

[batch:///opt/splunk/data/Test2PT]
index=main
sourcetype=testing2pt-kvp
move_policy=sinkhole
_TCP_ROUTING=pt01

---------- outputs.conf ----------

[tcpout]
defaultGroup=noforward
disabled=false

[indexAndForward]
index=true
selectiveIndexing=true

[tcpout:pt01]
server=PT01:9997


This move solved my requirement, that is indexed data locally on TEST01 and forward some data to PT01 as well but raised a new set of problems.
As mentioned our TEST01 is a standalone Splunk Enterprise installation, it has data being forwarded to it from UF on other hosts, scripted inputs, rest endpoints, etc. data inputs are configured which gets indexed daily over it. After adding this configuration all other data inputs apart from the one's mentioned in inputs.conf stopped indexing totally.

Can I get some help to reach the end as on how I can index literally all the data on TEST01 locally and just monitor forward files from one folder, in this case "/opt/splunk/data/Test2PT" to PT01 ?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...