Is it possible to store events coming from the same source in different indexes, depending on their content?

[michaje]

Hi,

Perhaps this question has been asked before...  Is it possible to store events coming from the same source in different indexes, depending on their content?

The use case is that some events are more sensitive than others and need to be sent to different indexes.

In our case, the index name would appear within the event, as a formatted field, like [index: SENSITIVE].

The input is a TCP port.

Any help would be appreciated, and I prefer to take no as an answer than to be led into some intricate solution.

Thank you,

Jean

[isoutamo]

Hi

it’s possible, but what is your real issue what you are solving this way?

How your stream is generated in source side and are there several or only one source?

r. Ismo

[michaje]

Thank you for the quick answer!

The question here is whether the <new-value> can be a variable found within the event using a regexp that would extract the value.

[dural_yyz]

I believe so but I've never tested and I don't have a dev environment to verify.  You can try inside your regex to create an unnamed capture group.  Inside the FORMAT tag replace <new-value> with "$1".

[michaje]

Thank you for the suggestion.  I could not test it, as an alternative approach has been adopted in the meantime.

[dural_yyz]

transforms.conf

 [index_reset]
 SOURCE_KEY = _raw
 DEST_KEY =  _MetaData:index
 REGEX = .
 FORMAT = index::<new-value>

This searches the _raw data feed for the regex match (change my example), then applies the FORMAT to the DEST_KEY.

Test in development environment first to fine tune this process, it can be tricky to get the regex and format just right.