Getting Data In
Highlighted

Event-based index routing at indexer layer when heavy forwarder is involved

Path Finder

I'm currently sending logs from a UF > HF > two indexer clusters.

I have the need to set the index name at the indexing layer, since the name of the index will be different, depending on the indexer cluster.

I tried putting the following props and transforms at the indexing layer:

props.conf:

[my:sourcetype]
TRANSFORMS-route_to_new_index = set_new_index

transforms.conf:

[set_new_index]
SOURCE_KEY = MetaData:Sourcetype
DEST_KEY =_MetaData:Index
REGEX = (sourcetype::my:sourcetype)
FORMAT = new_index

This does not change the index, however placing these same props and transforms on the HF does change the index. This doesn't help me though since the index name needs to be set after the data is split off to each indexer cluster. Is it really not possible to do this at the indexer layer when a HF is involved? Any other suggestions on how to accomplish the index rename?

Thanks

Highlighted

Re: Event-based index routing at indexer layer when heavy forwarder is involved

SplunkTrust
SplunkTrust

@ehowardl3 is the data split/cloned to both idx clusters or load balanced?

0 Karma
Highlighted

Re: Event-based index routing at indexer layer when heavy forwarder is involved

Path Finder

@MuS, thanks for your time. The data is split/cloned to both idx clusters.

0 Karma
Highlighted

Re: Event-based index routing at indexer layer when heavy forwarder is involved

SplunkTrust
SplunkTrust

please hold, this can be done on the HWF 😉

0 Karma
Highlighted

Re: Event-based index routing at indexer layer when heavy forwarder is involved

SplunkTrust
SplunkTrust

Hi ehowardl3,

first you need to have two separate tcpout entries in outputs.conf of your HWF:

[tcpout]
defaultGroup = cluster1 

[tcpout:cluster1]
server = <ip address>:<port>

[tcpout:cluster2]
server = <ip address>:<port>

Next you need a props.conf like the one you had with an additional line:

[my:sourcetype]
TRANSFORMS-001-route_to_new_index_cluster2 = 001-route_to_new_index_cluster2, 002-route_to_cluster2

then in transforms.conf you set it up like this:

[001-route_to_new_index_cluster2]
DEST_KEY =_MetaData:Index
REGEX = .
FORMAT = my_new_index # set to what ever you want

[002-route_to_cluster2]
REGEX = .
DEST_KEY = _TCP_ROUTING
FORMAT = cluster2  # which is the name from outputs.conf

The option defaultGroup=cluster1 in outputs.conf will send all data unchanged to cluster1.

After applying this config you need to restart the HWF.

Hope this helps ...

cheers, MuS

View solution in original post

Highlighted

Re: Event-based index routing at indexer layer when heavy forwarder is involved

Path Finder

Thank you! This makes sense and should work for me. So if there are two different source types that I need to send to two different indexes, I'm assuming the props and transforms would look something like this:

transforms.conf:

[001-route_to_new_index_cluster2]
SOURCE_KEY = MetaData:Sourcetype
DEST_KEY =_MetaData:Index
REGEX = (sourcetype::my:sourcetype1)
FORMAT = my_new_index1

[002-route_to_new_index_cluster2]
SOURCE_KEY = MetaData:Sourcetype
DEST_KEY =_MetaData:Index
REGEX = (sourcetype::my:sourcetype2)
FORMAT = my_new_index2


[003-route_to_cluster2]
REGEX = .
DEST_KEY = _TCP_ROUTING
FORMAT = cluster2  # which is the name from outputs.conf

props.conf:

[my:sourcetype1]
TRANSFORMS-001-route_to_new_index_cluster2 = 001-route_to_new_index_cluster2, 003-route_to_cluster2

[my:sourcetype2]
TRANSFORMS-002-route_to_new_index_cluster2 = 002-route_to_new_index_cluster2, 003-route_to_cluster2

Correct?

Thanks!

0 Karma
Highlighted

Re: Event-based index routing at indexer layer when heavy forwarder is involved

SplunkTrust
SplunkTrust

yep, that should work

0 Karma
Highlighted

Re: Event-based index routing at indexer layer when heavy forwarder is involved

SplunkTrust
SplunkTrust

except the REGEX, just use . because the props.conf already limits the use of the transforms to specific sourcetypes 😉

0 Karma
Highlighted

Re: Event-based index routing at indexer layer when heavy forwarder is involved

Path Finder

Ah yeah, good point. Thanks again!

0 Karma
Highlighted

Re: Event-based index routing at indexer layer when heavy forwarder is involved

Path Finder

@MuS, one thing that has me a little confused - since props.conf calls out the sourcetype and then routes it to cluster2, won't that catch all the data of that sourcetype instead of splitting the data and sending it to cluster2 and to the default group?

0 Karma