topic Re: Forward data to two different indexers and filter a field out in Getting Data In

Forward data to two different indexers and filter a field out

klim — Fri, 06 Dec 2024 00:29:46 GMT

I need to forward data from a heavy forwarder to two different indexer clusters. One of the clusters needs to have a field removed. If I use sedcmd in props.conf on the HF it removes it for both and putting sedcmd in props.conf on one of the indexers doesn't work (it does work if i bypass the HF).

Is there a way to do this?

Edit: I was thinking of using an intermediate forwarder so heavy forwarder -> another heavy forwarder -> indexer cluster but the intermediate heavy forwarder props.conf does not work.

Re: Forward data to two different indexers and filter a field out

PaulPanther — Fri, 06 Dec 2024 06:58:44 GMT

Try to set

sendCookedData=false

for the second HF output in your outputs.conf and then apply your props.conf on your second HF.

Re: Forward data to two different indexers and filter a field out

PickleRick — Fri, 06 Dec 2024 09:19:55 GMT

This will actually send raw data suitable to further processing by third party solution. It will not keep the metadata, it will not use s2s protocol, just send "TCP syslog" stream.

Re: Forward data to two different indexers and filter a field out

PickleRick — Fri, 06 Dec 2024 09:24:14 GMT

What is it with the latest peak of question about "sending the data into two indexer(s| clusters) while modifying one stream"? Suddenly everyone has this borderline use case?

Why do that in the first place? Is it really worth paying extra for double the license? What actually is your use case?