I'm currently sending logs from a UF > HF > two indexer clusters.
I have the need to set the index name at the indexing layer, since the name of the index will be different, depending on the indexer cluster.
I tried putting the following props and transforms at the indexing layer:
[my:sourcetype] TRANSFORMS-route_to_new_index = set_new_index
[set_new_index] SOURCE_KEY = MetaData:Sourcetype DEST_KEY =_MetaData:Index REGEX = (sourcetype::my:sourcetype) FORMAT = new_index
This does not change the index, however placing these same props and transforms on the HF does change the index. This doesn't help me though since the index name needs to be set after the data is split off to each indexer cluster. Is it really not possible to do this at the indexer layer when a HF is involved? Any other suggestions on how to accomplish the index rename?
first you need to have two separate
tcpout entries in
outputs.conf of your HWF:
[tcpout] defaultGroup = cluster1 [tcpout:cluster1] server = <ip address>:<port> [tcpout:cluster2] server = <ip address>:<port>
Next you need a
props.conf like the one you had with an additional line:
[my:sourcetype] TRANSFORMS-001-route_to_new_index_cluster2 = 001-route_to_new_index_cluster2, 002-route_to_cluster2
transforms.conf you set it up like this:
[001-route_to_new_index_cluster2] DEST_KEY =_MetaData:Index REGEX = . FORMAT = my_new_index # set to what ever you want [002-route_to_cluster2] REGEX = . DEST_KEY = _TCP_ROUTING FORMAT = cluster2 # which is the name from outputs.conf
outputs.conf will send all data unchanged to
After applying this config you need to restart the HWF.
Hope this helps ...
Thank you! This makes sense and should work for me. So if there are two different source types that I need to send to two different indexes, I'm assuming the props and transforms would look something like this:
[001-route_to_new_index_cluster2] SOURCE_KEY = MetaData:Sourcetype DEST_KEY =_MetaData:Index REGEX = (sourcetype::my:sourcetype1) FORMAT = my_new_index1 [002-route_to_new_index_cluster2] SOURCE_KEY = MetaData:Sourcetype DEST_KEY =_MetaData:Index REGEX = (sourcetype::my:sourcetype2) FORMAT = my_new_index2 [003-route_to_cluster2] REGEX = . DEST_KEY = _TCP_ROUTING FORMAT = cluster2 # which is the name from outputs.conf
[my:sourcetype1] TRANSFORMS-001-route_to_new_index_cluster2 = 001-route_to_new_index_cluster2, 003-route_to_cluster2 [my:sourcetype2] TRANSFORMS-002-route_to_new_index_cluster2 = 002-route_to_new_index_cluster2, 003-route_to_cluster2
REGEX, just use
. because the
props.conf already limits the use of the transforms to specific sourcetypes 😉
@MuS, one thing that has me a little confused - since props.conf calls out the sourcetype and then routes it to cluster2, won't that catch all the data of that sourcetype instead of splitting the data and sending it to cluster2 and to the default group?