topic Re: Sourcetype cloning - logs stopped in Getting Data In

Sourcetype cloning - logs stopped

Nicolas2203 — Mon, 08 Apr 2024 15:57:14 GMT

Hi all

After temptative for troubleshooting my issue alone, I will try my luck here.

Purpose : clone one sourcetype to store the logs into a local indexer, and in a distant one
I use one heavy forwarder to receive the logs, store the logs in a indexer, and same heavy forwarder will clone the sourcetype to forward the cloned one into a distant heavy forward, that I don't managed.
Here is my config :

[inputs.conf] [udp://22210] index = my_logs_indexer sourcetype = log_sourcetype disabled = false

This works pretty well, and all logs are stored into my indexer

Now will come the cloning part :

[props.conf] [log_sourcetype] TRANSFORMS-log_sourcetype-clone = log_sourcetype-clone [transforms.conf] [log_sourcetype-clone] REGEX = . DEST_KEY = _TCP_ROUTING FORMAT = distant_HF_formylogs [outputs.conf] => for cloned logs [tcpout:distant_HF_formylogs] server = ip_of_distant_HF:port sendCookedData = false

This configuration is used for another use case, as sometimes I have had to anonymize some logs. However, for this particular use case, when I activate the cloning part, it stops the complete log flow for this use case, even on the local indexers. I didn't quite understand why, because I don't see the difference with my other use case, apart from the fact that the logs are UDP logs and not TCP. Am I missing something?

Thanks a lot for your help

Re: Sourcetype cloning - logs stopped

PickleRick — Mon, 08 Apr 2024 18:51:22 GMT

When you're overwriting the value of _TCP_ROUTING metadata field, you're effectively telling Splunk to route the events to this destination (output group) only.

If you want to route some data to more than one output group, you must include all relevant output groups in _TCP_ROUTING. Like

_TCP_ROUTING = my_primary_indexers, my_secondary_indexers

Read the https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Configure_routing Of course you don't have to put the transforms.conf into etc/system/local (in fact it'd be best if you didn't do that).

Re: Sourcetype cloning - logs stopped

Nicolas2203 — Tue, 09 Apr 2024 08:08:44 GMT

Hi PickleRick, thanks for your response and time.

The cloned logs are routing only to one instance, specified into the outputs.conf

The "original" logs, not the cloned one are directed to my local indexers, just the cloned sourcetype is directed with another heavy forwarder specified in the outputs.conf placed in the same app as the props and transforms.

Not sure if i'm clear 😕

Re: Sourcetype cloning - logs stopped

PickleRick — Tue, 09 Apr 2024 09:03:02 GMT

Yes. I understand. They are not "cloned", they are redirected.

The events are sent to _all_ output groups specified in the outputs.conf (or to the specified output group(s), if you manipulated _TCP_ROUTING manually). Within each of applicable group the event is sent to just one of the servers configured in such group.

So you must make sure that the events you want have both output groups specified in _TCP_ROUTING.

Re: Sourcetype cloning - logs stopped

Nicolas2203 — Tue, 09 Apr 2024 17:20:12 GMT

Ok I understand what you say. But sorry I forgot to mentionned that I have a TCPOUT default on my conf:

[tcpout] defaultGroup = my_indexers forceTimebasedAutoLB = true forwardedindex.filter.disable = true [tcpout:my_indexers] server = indexer1:9997, indexer2:9997

So if I'am correct, the inputs.conf :

[inputs.conf] [udp://22210] index = my_logs_indexer sourcetype = log_sourcetype disabled = false

Redirect the logs to the default outputs, because no outputs is specified .

Correct me if I'm wrong, and sorry to forgot this config at the first question

Re: Sourcetype cloning - logs stopped

PickleRick — Tue, 09 Apr 2024 11:48:27 GMT

Yes. The [tcpout] defaultGroup setting tells your Splunk component what to do with events by default. So if you don't modify the _TCP_ROUTING field, your events should be going to the my_indexers group.

But when you overwrite the _TCP_ROUTING with just distant_HF_formylogs, you'll be sending to that group only.

Re: Sourcetype cloning - logs stopped

Nicolas2203 — Thu, 11 Apr 2024 13:16:14 GMT

Ok, to be honest I had to check more on the config to properly clone and forward my datas, the behaviour of the conf it's strange. But thanks a lot for your help, I appreciate !