Getting Data In

Can the universal forwarder send the same data to different farms with different index names?

ddrillic
Ultra Champion

We are in a transition from the "legacy" farm to the new one. During this transition period, the clients would like to send the same data to different farms with different index names. Is it possible?

After all the outputs.conf files get combined...

Tags (1)
0 Karma
1 Solution

micahkemp
Champion

I think your best bet may be to have your universal forwarder(s) send to two heavy forwarders thusly:

[tcpout]
defaultGroup=heavyforwarder1,heavyforwarder2

[tcpout:heavyforwarder1]
server=10.1.1.197:9997

[tcpout:heavyforwarder2]
server=10.1.1.200:9997

Each heavy forwarder would be configured to rewrite _MetaData:Index appropriately via this type of config:

props.conf:

[default]
TRANSFORMS-sendalltocorrectindex = sendalltocorrectindex

transforms.conf:

[sendalltocorrectindex]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = index1

(with heavyforwarder2 using different value for FORMAT to specify the valid index for that site/farm/whatever)

The key is that the duplicate feeding needs to occur at the Universal Forwarder, specifically because UFs don't cook data. Once the data is cooked, it stays cooked, and additional indexers, heavy forwarders, whatever won't do any further props/transforms on it (that's not 100% true, but it's an exception and ugly configuration to make it false).

View solution in original post

micahkemp
Champion

I think your best bet may be to have your universal forwarder(s) send to two heavy forwarders thusly:

[tcpout]
defaultGroup=heavyforwarder1,heavyforwarder2

[tcpout:heavyforwarder1]
server=10.1.1.197:9997

[tcpout:heavyforwarder2]
server=10.1.1.200:9997

Each heavy forwarder would be configured to rewrite _MetaData:Index appropriately via this type of config:

props.conf:

[default]
TRANSFORMS-sendalltocorrectindex = sendalltocorrectindex

transforms.conf:

[sendalltocorrectindex]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = index1

(with heavyforwarder2 using different value for FORMAT to specify the valid index for that site/farm/whatever)

The key is that the duplicate feeding needs to occur at the Universal Forwarder, specifically because UFs don't cook data. Once the data is cooked, it stays cooked, and additional indexers, heavy forwarders, whatever won't do any further props/transforms on it (that's not 100% true, but it's an exception and ugly configuration to make it false).

ddrillic
Ultra Champion

Gorgeous solution @micahkemp. Took us a couple of months to test it ; - )

It's interesting about the underscore difference between DEST_KEY = _MetaData:Index and DEST_KEY = MetaData:Host.

0 Karma

ddrillic
Ultra Champion

Very interesting @micahkemp!!

Maybe I completely miss the point but let's say I have two apps in the UF, each monitors the same data, in the inputs.conf we'll specify the specific index names and in each app's outputs.conf we'll specify the indexers of the proper farm.

Will it work?

0 Karma

micahkemp
Champion

You’re going to have issues monitoring the same file twice. Splunk will realize it’s already been indexed and skip it.

ddrillic
Ultra Champion

Gorgeous @micahkemp! Is there a way to avoid it? or "just" two forwarders on the same server...

0 Karma

micahkemp
Champion

I’m not saying this is the only way, but it’s the only way I could think of.

Maybe another option would be to have a [default] props entry pointing to a transform to change the index on the indexers. If you go this route, maybe you do this on the old indexers and change your inputs on the forwarders to point to the new index. This way you leave your complexity on the indexers that will be retired.

ddrillic
Ultra Champion

Makes perfect sense @micahkemp. This should do it.

0 Karma

micahkemp
Champion

You could do all of the posted configuration on the indexers, I just kinda assumed you didn’t want to due to the nature of your question. The configs should be helpful either way, the key is the events need to be sent from a Universal Forwarder to two different Splunk instanes, which is where the index change occurs.

0 Karma

ddrillic
Ultra Champion

Perfect - thank you.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...