Getting Data In

How to configure a heavy forwarder to reroute data from a universal forwarder into a special index different from the one specified in the UF's inputs.conf?

tpaulsen
Contributor

Hi,

We need some help with rerouting data from a universal forwarder via a heavy forwarder into a different index than given in the UF inputs.conf.

Universal Forwarder inputs.conf:

[monitor:///var/opt/log/tomcat/test.log]
disabled = false
sourcetype = testdatenb2c

Heavy Forwarder (as man in the middle) configuration:
inputs.conf:

[default]
host = splunk-heavy-dev-369368
[splunktcp://9697]
connection_host = ip

props.conf:

[testdatenb2c]
TRANSFORMS-routing = transforms_TEST

transforms.conf

[transforms_TEST]
DEST_KEY=_TCP_ROUTING
FORMAT=B2CDATEN2EC
REGEX=.

outputs.conf

[tcpout:B2CDATEN2EC]
#Disable the current filters from the defaults outputs.conf
forwardedindex.0.whitelist =
forwardedindex.1.blacklist =
forwardedindex.2.whitelist =

forwardedindex.0.whitelist = test
disabled = false
server = my-splunk-idx-dev:9997

Now, the data gets indexed by the my-splunk-idx-dev server, but the data doesn't land in the index "test". Instead, it is being indexed in "main" as the default configured on the UF.
What are we doing wrong here? How can we reroute the data to our "test" index on the my-splunk-idx-dev server?

Thank you in advance for any help,

Best regards, Tom

1 Solution

somesoni2
Revered Legend

The outputs.conf forwardedindex whitelist setting only does the filter of events based on index name but doesn't update the index name in the data. That has to be done in the props-transforms conf file.

Try this for your props.conf and transforms.conf in your Heavy forwarder
Updated fixed typo, added missing underscore

props.conf
[testdatenb2c]
TRANSFORMS-routing = transforms_routing,transforms_index

transforms.conf

[transforms_routing]
DEST_KEY=_TCP_ROUTING
FORMAT=B2CDATEN2EC
REGEX=.

[transforms_index]
DEST_KEY = _MetaData:Index
FORMAT = test
REGEX=.

View solution in original post

somesoni2
Revered Legend

The outputs.conf forwardedindex whitelist setting only does the filter of events based on index name but doesn't update the index name in the data. That has to be done in the props-transforms conf file.

Try this for your props.conf and transforms.conf in your Heavy forwarder
Updated fixed typo, added missing underscore

props.conf
[testdatenb2c]
TRANSFORMS-routing = transforms_routing,transforms_index

transforms.conf

[transforms_routing]
DEST_KEY=_TCP_ROUTING
FORMAT=B2CDATEN2EC
REGEX=.

[transforms_index]
DEST_KEY = _MetaData:Index
FORMAT = test
REGEX=.

View solution in original post

tpaulsen
Contributor

Thank you very much! It worked! For others...note that it´s _MetaData (with leading underscore).

0 Karma

wkupersa
Path Finder

Thanks to both of you. Very helpful!

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!