Using a transform to set index from event field. How to fall back to a certain index?

[juniormint]

I have events with a field that contains a desired destination index (see index=* below).

[timestamp] index=layer1 message="123456"
[timestamp] index=layer2 message="123456"
[timestamp] index=layer3 message="123456"
[timestamp] index=jumbled message="123456"
[timestamp] index=layer1 message="123456"
[timestamp] index=layer1 message="123456"
[timestamp] index=layer2 message="123456"

I'm currently using a transform like the following, and it works quite well to get events into their correct index.

[MyTransform]
REGEX = .*index="(.*?)"
DEST_KEY = _MetaData:Index
FORMAT = $1

At least most of the time. Occasionally an event arrives whose index field names an index which doesn't exist (either because the name is wrong or I forgot to create the index). In this case the above transform appears to simply generate an error in the internal index and otherwise drop the data.

Is there another way to do this that would drop messages which failed match an existing index into some catch all index of my choosing?

Something like FORMAT if ($1).exists $1 else myDefaultCatchAllIndex

[woodcock]

You need lastChanceIndex but this is a global setting; this is as close as you can get:
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Indexesconf#GLOBAL_SETTINGS

[woodcock]

I would use INGEST_EVAL to do this:

INGEST_EVAL = index = replace(_raw, some stuff here, "")

[tyron_]

How does that solve the problem of evaluating if an index exists and, if not, fallback to a default index?

[woodcock]

OH, I see what you mean. See my new answer.

[_d_]

You can create another transforms.conf stanza that captures everything and call it in order.

transforms.conf

[MyTransform] 
REGEX = .index="(.?)" 
DEST_KEY = _MetaData:Index 
FORMAT = $1

[MyTransform_all] 
REGEX = . 
DEST_KEY = _MetaData:Index 
FORMAT = catchAll

props.conf

[mysourcetype]
TRANSFORMS-index_routing = MyTransform_all, MyTransform

[tyron_]

6 years later, is this still not possible?

[_d_]

You cannot simply define an index on the fly, if that's what you're asking.

[kristian_kolb]

Yes it matters a lot.

essentially, the TRANSFORMS directive in props.conf sends all of your events on a cruise to two magic islands before going on to Index heaven; On the first island, any event will have it's destination index set to 'catchAll', then it leaves for the second island. There some leprechauns will inspect them and perform some regex magic rites, and only those that have a clear knowledge of where they want to go, will be allowed to do so, i.e. the leprechauns will set their index to the $1 capture group.

When the events come back to dry land, they go to the configured index.

[juniormint]

does the order of listing them matter? should it be

[mysourcetype] TRANSFORMS-index_routing = MyTransform, MyTransform_all

[kristian_kolb]

This would work for garbled events, where no match can be made for 'index=someindex', but it will not work any better for events that are properly read, but contains an unconfigured index as the field value.

Unfortunately, I don't think there is any way to create any 'if-exist' logic at that point....