Sourcetype cloning - logs stopped

[Nicolas2203]

Hi all

After temptative for troubleshooting my issue alone, I will try my luck here.

Purpose : clone one sourcetype to store the logs into a local indexer, and in a distant one
I use one heavy forwarder to receive the logs, store the logs in a indexer, and same heavy forwarder will clone the sourcetype to forward the cloned one into a distant heavy forward, that I don't managed.
Here is my config :

[inputs.conf]
[udp://22210]
index = my_logs_indexer
sourcetype = log_sourcetype
disabled = false

This works pretty well, and all logs are stored into my indexer

Now will come the cloning part : 

[props.conf]
[log_sourcetype]
TRANSFORMS-log_sourcetype-clone = log_sourcetype-clone

[transforms.conf]
[log_sourcetype-clone]
REGEX = .
DEST_KEY = _TCP_ROUTING
FORMAT = distant_HF_formylogs

[outputs.conf] => for cloned logs
[tcpout:distant_HF_formylogs]
server = ip_of_distant_HF:port
sendCookedData = false

This configuration is used for another use case, as sometimes I have had to anonymize some logs. However, for this particular use case, when I activate the cloning part, it stops the complete log flow for this use case, even on the local indexers. I didn't quite understand why, because I don't see the difference with my other use case, apart from the fact that the logs are UDP logs and not TCP. Am I missing something?

Thanks a lot for your help

[Nicolas2203]

Ok, to be honest I had to check more on the config to properly clone and forward my datas, the behaviour of the conf it's strange. But thanks a lot for your help, I appreciate !

[Nicolas2203]

Ok I understand what you say. But sorry I forgot to mentionned that I have a TCPOUT default on my conf:

[tcpout]
defaultGroup = my_indexers
forceTimebasedAutoLB = true
forwardedindex.filter.disable = true

[tcpout:my_indexers]
server = indexer1:9997, indexer2:9997

So if I'am correct, the inputs.conf :

[inputs.conf]
[udp://22210]
index = my_logs_indexer
sourcetype = log_sourcetype
disabled = false

Redirect the logs to the default outputs, because no outputs is specified .

Correct me if I'm wrong, and sorry to forgot this config at the first question

[PickleRick]

Yes. The [tcpout] defaultGroup setting tells your Splunk component what to do with events by default. So if you don't modify the _TCP_ROUTING field, your events should be going to the my_indexers group.

But when you overwrite the _TCP_ROUTING with just distant_HF_formylogs, you'll be sending to that group only.

[Nicolas2203]

Hi PickleRick, thanks for your response and time.

The cloned logs are routing only to one instance, specified into the outputs.conf

The "original" logs, not the cloned one are directed to my local indexers, just the cloned sourcetype is directed with another heavy forwarder specified in the outputs.conf placed in the same app as the props and transforms.

Not sure if i'm clear 😕

[PickleRick]

Yes. I understand. They are not "cloned", they are redirected.

The events are sent to _all_ output groups specified in the outputs.conf (or to the specified output group(s), if you manipulated _TCP_ROUTING manually). Within each of applicable group the event is sent to just one of the servers configured in such group.

So you must make sure that the events you want have both output groups specified in _TCP_ROUTING.

[PickleRick]

When you're overwriting the value of _TCP_ROUTING metadata field, you're effectively telling Splunk to route the events to this destination (output group) only.

If you want to route some data to more than one output group, you must include all relevant output groups in _TCP_ROUTING. Like

_TCP_ROUTING = my_primary_indexers, my_secondary_indexers

Read the https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Configure_routing Of course you don't have to put the transforms.conf into etc/system/local (in fact it'd be best if you didn't do that).