topic Re: Can the universal forwarder send the same data to different farms with different index names? in Getting Data In

Can the universal forwarder send the same data to different farms with different index names?

ddrillic — Fri, 09 Feb 2018 18:17:15 GMT

We are in a transition from the "legacy" farm to the new one. During this transition period, the clients would like to send the same data to different farms with different index names. Is it possible?

After all the outputs.conf files get combined...

Re: Can the universal forwarder send the same data to different farms with different index names?

micahkemp — Sat, 10 Feb 2018 03:16:18 GMT

I think your best bet may be to have your universal forwarder(s) send to two heavy forwarders thusly:

[tcpout]
defaultGroup=heavyforwarder1,heavyforwarder2

[tcpout:heavyforwarder1]
server=10.1.1.197:9997

[tcpout:heavyforwarder2]
server=10.1.1.200:9997

Each heavy forwarder would be configured to rewrite _MetaData:Index appropriately via this type of config:

props.conf:

[default]
TRANSFORMS-sendalltocorrectindex = sendalltocorrectindex

transforms.conf:

[sendalltocorrectindex]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = index1

(with heavyforwarder2 using different value for FORMAT to specify the valid index for that site/farm/whatever)

The key is that the duplicate feeding needs to occur at the Universal Forwarder, specifically because UFs don't cook data. Once the data is cooked, it stays cooked, and additional indexers, heavy forwarders, whatever won't do any further props/transforms on it (that's not 100% true, but it's an exception and ugly configuration to make it false).

Re: Can the universal forwarder send the same data to different farms with different index names?

ddrillic — Sat, 10 Feb 2018 22:24:14 GMT

Very interesting @micahkemp!!

Maybe I completely miss the point but let's say I have two apps in the UF, each monitors the same data, in the inputs.conf we'll specify the specific index names and in each app's outputs.conf we'll specify the indexers of the proper farm.

Will it work?

Re: Can the universal forwarder send the same data to different farms with different index names?

micahkemp — Sat, 10 Feb 2018 22:37:41 GMT

You’re going to have issues monitoring the same file twice. Splunk will realize it’s already been indexed and skip it.

Re: Can the universal forwarder send the same data to different farms with different index names?

ddrillic — Sat, 10 Feb 2018 23:00:12 GMT

Gorgeous @micahkemp! Is there a way to avoid it? or "just" two forwarders on the same server...

Re: Can the universal forwarder send the same data to different farms with different index names?

micahkemp — Sat, 10 Feb 2018 23:06:09 GMT

I’m not saying this is the only way, but it’s the only way I could think of.

Maybe another option would be to have a [default] props entry pointing to a transform to change the index on the indexers. If you go this route, maybe you do this on the old indexers and change your inputs on the forwarders to point to the new index. This way you leave your complexity on the indexers that will be retired.

Re: Can the universal forwarder send the same data to different farms with different index names?

ddrillic — Sat, 10 Feb 2018 23:09:14 GMT

Makes perfect sense @micahkemp. This should do it.

Re: Can the universal forwarder send the same data to different farms with different index names?

micahkemp — Sat, 10 Feb 2018 23:46:04 GMT

You could do all of the posted configuration on the indexers, I just kinda assumed you didn’t want to due to the nature of your question. The configs should be helpful either way, the key is the events need to be sent from a Universal Forwarder to two different Splunk instanes, which is where the index change occurs.

Re: Can the universal forwarder send the same data to different farms with different index names?

ddrillic — Sun, 11 Feb 2018 00:38:00 GMT

Perfect - thank you.

Re: Can the universal forwarder send the same data to different farms with different index names?

ddrillic — Wed, 08 Aug 2018 19:04:32 GMT

Gorgeous solution @micahkemp. Took us a couple of months to test it ; - )

It's interesting about the underscore difference between DEST_KEY = _MetaData:Index and DEST_KEY = MetaData:Host.