Route index data based on source

[rreatiga]

Hi,

Is it possible using props.conf and transforms.conf to route some data on an index based on the source field?

Let's say index1 contains a lot of sources, in some sources it contains certain words in the path for example

(source="*dev-ksm*" OR source="*int-ksm*" OR source="*qa-ksm*" OR source="*amq-*-ksm*")

For this scenario I'd like to route events that their source contains the above matching sources to an index2

Was thinking in something like this:

props.conf

[index::current_index]
TRANSFORMS-routing=filter-to-new_index

transforms.conf

[filter-to-new_index]
DEST_KEY = _MetaData:Index 
SOURCE_KEY = MetaData:Source 
REGEX = (?i)(.*dev-ksm.*|.*int-ksm.*|.*qa-ksm.*|.*amq-.*-ksm.*)
FORMAT = new_index

Does not seem to be currently working. Hence the question if its possible to do something like this.

Thanks in advance.

[gcusello]

Hi @rreatiga,

in the header of the props.conf stanza, you cannot use "index::current_index" but only sourcetype or source or host fields.

for source and host, you can also use jolly char, something like this:

props.conf

[source::*dev-ksm*]
TRANSFORMS-routing=filter-to-new_index

[source::*int-ksm*]
TRANSFORMS-routing=filter-to-new_index

[source::*qa-ksm*]
TRANSFORMS-routing=filter-to-new_index

[source::*amq-*-ksm*]
TRANSFORMS-routing=filter-to-new_index

transorms.conf

[filter-to-new_index]
DEST_KEY = _MetaData:Index 
REGEX = .
FORMAT = new_index

Ciao.

Giuseppe