I have events with a field that contains a desired destination index (see index=* below).
[timestamp] index=layer1 message="123456"
[timestamp] index=layer2 message="123456"
[timestamp] index=layer3 message="123456"
[timestamp] index=jumbled message="123456"
[timestamp] index=layer1 message="123456"
[timestamp] index=layer1 message="123456"
[timestamp] index=layer2 message="123456"
I'm currently using a transform like the following, and it works quite well to get events into their correct index.
[MyTransform]
REGEX = .*index="(.*?)"
DEST_KEY = _MetaData:Index
FORMAT = $1
At least most of the time. Occasionally an event arrives whose index field names an index which doesn't exist (either because the name is wrong or I forgot to create the index). In this case the above transform appears to simply generate an error in the internal index and otherwise drop the data.
Is there another way to do this that would drop messages which failed match an existing index into some catch all index of my choosing?
Something like FORMAT if ($1).exists $1 else myDefaultCatchAllIndex
