Is there a limitation in the combination of transforms on a source in props.conf?

[harald_leitl]

Hi all,

is there a limitation in the combination of transforms on a source in props.conf?

here is what i did and somehow I don't get any result.

Whenever I delete the TRANSFORMS-reroute entry, data is received and hostnames are changed.

Somehow I don't get the source with my regex rerouted to another index.

props.conf:

[source::tcp:514]
TRUNCATE = 64000
TRANSFORMS = newhost1
TRANSFORMS = newhost2
TRANSFORMS-reroute=set-index

transforms.conf

[newhost1]
DEST_KEY = MetaData:Host   
REGEX = mymatchinghost1rex 
FORMAT = host::myhost1

[newhost2]
DEST_KEY = MetaData:Host
REGEX = mymatchinghost2rex
FORMAT = host::myhost2

[set-index]
DEST_KEY=_MetaData:Index
REGEX= .+mymatchingrex.+
FORMAT=myindex
WRITE_META=true

thanks for your help,

kind regards,

harald

[yeahnah · Accepted solution]

Hi @harald_leitl 

I'm not aware of a limitation and, if there is one, it is probably a very large number.

As per the spec, all the props.conf entries should use the TRANSFORMS-<class> notation so maybe that is an issue - it kinda works when the set-index entry is removed by luck.

This should work 

[source::tcp:514]
TRUNCATE = 64000
TRANSFORMS-hostrename = newhost1, newhost2
TRANSFORMS-reroute = set-index

For the transforms.conf set-index entry I do not think the WRITE_META parameter is needed as it's not a index-time field extraction

WRITE_META = <boolean>
* NOTE: This setting is only valid for index-time field extractions.
* Automatically writes REGEX to metadata.
* Required for all index-time field extractions except for those where
  DEST_KEY = _meta (see the description of the DEST_KEY setting, below)
* Use instead of DEST_KEY = _meta.
* Default: false

The .+ is not really necessary at the start/end of the REGEX either, but it should also do no harm.  Anyway, try this and see if it helps.

[set-index]
DEST_KEY = _MetaData:Index
REGEX = mymatchingrex
FORMAT = myindex
#WRITE_META = true

 Restart the test heavy forwarder/indexer to ensure the configs are applied.

Hope this helps.

[harald_leitl]

Hi @yeahnah,

That did the trick.

It's working now...

 hosts are renamed and data is rerouted into the correct index.

Thanks a lot,

cheers