Getting Data In

Receive data on intermediate or heavy forwarder, consolidate into one index, and forward to single index externally

Path Finder

Hi all, for the last few months I've been working on a splunk environment and need some assistance with routing data.

The current setup:
Universal forwarders cloning data to indexer cluster and intermediate forwarder on roughly 20 different indexes.
Heavy Forwarder with AWS addon forwarding local and AWS cloning data to different indexes on indexers and intermediate forwarder.

What we want to do is send the data to the indexers like normal and send all of the cloned data being received by the intermediate forwarder to a single index on an external splunk enterprise indexer.

I have tried changing the intermediate to a heavy forwarder (locally indexing the data into their respective indexes) and forwarding to the external party to a single index, but because of the amount of indexes, we need to consolidate the cloned data to a single index and forward,

I attempted using the transforms and props confs on the intermediate (now heavy) forwarder to change the receiving index for all data received on [splunktcp://9997] to go to a single index either locally and forward or to just forward to a single index. Below are examples of the confs on the intermediate. Note: I refer to it as the intermediate as that is what we want it to be, but currently have indexing setup on it.

Our long term goal is to use the intermediate to receive all of our data into a single index and index from our other environments into their own respective indexes, and forward to the external splunk environment.


compressed = true
disabled = 0


TRANSFORMS-forwarding = changeIndex, tcpForward


#SOURCE_KEY = _MetaData:Index
DEST_KEY = _MetaData:Index
FORMAT = test_index

FORMAT = external-splunk

Any help would be greatly appreciated. Currently with this and various setups, I'm not getting any errors, but the data is not getting to the external splunk environment.

0 Karma

Re: Receive data on intermediate or heavy forwarder, consolidate into one index, and forward to single index externally

Super Champion

So I assume the ultimate aim is to forward all data to the external-splunk ? Then you don't need to index at all in your HF or old Splunk. Just do this using HF on the fly

But before you do, please check if you are getting data into test_index. This will ensure there is no issues with network connectivity or data flow.
1) So if you need to send everything to external splunk, you don't need transforms. Just below settings is good

TRANSFORMS-forwarding = tcpForward



server = <external_Server>:<external_port>
sendCookedData = false

2) But if you really need to index for something reason and then forward.
You need to retain props.conf & transforms.conf as you have already written, but just add outputs.conf like below



Also good to have a look into

View solution in original post

0 Karma