Route index data based on source

rreatiga — Tue, 28 May 2024 22:11:43 GMT

Hi,

Is it possible using props.conf and transforms.conf to route some data on an index based on the source field?

Let's say index1 contains a lot of sources, in some sources it contains certain words in the path for example

(source="*dev-ksm*" OR source="*int-ksm*" OR source="*qa-ksm*" OR source="*amq-*-ksm*")

For this scenario I'd like to route events that their source contains the above matching sources to an index2

Was thinking in something like this:

props.conf

[index::current_index] TRANSFORMS-routing=filter-to-new_index

transforms.conf

[filter-to-new_index] DEST_KEY = _MetaData:Index SOURCE_KEY = MetaData:Source REGEX = (?i)(.*dev-ksm.*|.*int-ksm.*|.*qa-ksm.*|.*amq-.*-ksm.*) FORMAT = new_index

Does not seem to be currently working. Hence the question if its possible to do something like this.

Thanks in advance.

Re: Route index data based on source

gcusello — Wed, 29 May 2024 05:17:16 GMT

Hi @rreatiga,

in the header of the props.conf stanza, you cannot use "index::current_index" but only sourcetype or source or host fields.

for source and host, you can also use jolly char, something like this:

props.conf

[source::*dev-ksm*] TRANSFORMS-routing=filter-to-new_index [source::*int-ksm*] TRANSFORMS-routing=filter-to-new_index [source::*qa-ksm*] TRANSFORMS-routing=filter-to-new_index [source::*amq-*-ksm*] TRANSFORMS-routing=filter-to-new_index

transorms.conf

[filter-to-new_index] DEST_KEY = _MetaData:Index REGEX = . FORMAT = new_index

Ciao.

Giuseppe

topic Route index data based on source in Getting Data In

Route index data based on source

Re: Route index data based on source