Splunk Search

Using a transform to set index from event field. How to fall back to a certain index?

juniormint
Communicator

I have events with a field that contains a desired destination index (see index=* below).

[timestamp] index=layer1 message="123456"
[timestamp] index=layer2 message="123456"
[timestamp] index=layer3 message="123456"
[timestamp] index=jumbled message="123456"
[timestamp] index=layer1 message="123456"
[timestamp] index=layer1 message="123456"
[timestamp] index=layer2 message="123456"

I'm currently using a transform like the following, and it works quite well to get events into their correct index.

[MyTransform]
REGEX = .*index="(.*?)"
DEST_KEY = _MetaData:Index
FORMAT = $1

At least most of the time. Occasionally an event arrives whose index field names an index which doesn't exist (either because the name is wrong or I forgot to create the index). In this case the above transform appears to simply generate an error in the internal index and otherwise drop the data.

Is there another way to do this that would drop messages which failed match an existing index into some catch all index of my choosing?

Something like FORMAT if ($1).exists $1 else myDefaultCatchAllIndex

Tags (3)

woodcock
Esteemed Legend

You need lastChanceIndex but this is a global setting; this is as close as you can get:
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Indexesconf#GLOBAL_SETTINGS

woodcock
Esteemed Legend

I would use INGEST_EVAL to do this:

INGEST_EVAL = index = replace(_raw, some stuff here, "")
0 Karma

tyron_
Explorer

How does that solve the problem of evaluating if an index exists and, if not, fallback to a default index?

0 Karma

woodcock
Esteemed Legend

OH, I see what you mean. See my new answer.

0 Karma

_d_
Splunk Employee
Splunk Employee

You can create another transforms.conf stanza that captures everything and call it in order.

transforms.conf

[MyTransform] 
REGEX = .index="(.?)" 
DEST_KEY = _MetaData:Index 
FORMAT = $1

[MyTransform_all] 
REGEX = . 
DEST_KEY = _MetaData:Index 
FORMAT = catchAll

props.conf

[mysourcetype]
TRANSFORMS-index_routing = MyTransform_all, MyTransform

tyron_
Explorer

6 years later, is this still not possible?

0 Karma

_d_
Splunk Employee
Splunk Employee

You cannot simply define an index on the fly, if that's what you're asking.

0 Karma

kristian_kolb
Ultra Champion

Yes it matters a lot.

essentially, the TRANSFORMS directive in props.conf sends all of your events on a cruise to two magic islands before going on to Index heaven; On the first island, any event will have it's destination index set to 'catchAll', then it leaves for the second island. There some leprechauns will inspect them and perform some regex magic rites, and only those that have a clear knowledge of where they want to go, will be allowed to do so, i.e. the leprechauns will set their index to the $1 capture group.

When the events come back to dry land, they go to the configured index.

juniormint
Communicator

does the order of listing them matter? should it be

[mysourcetype] TRANSFORMS-index_routing = MyTransform, MyTransform_all

0 Karma

kristian_kolb
Ultra Champion

This would work for garbled events, where no match can be made for 'index=someindex', but it will not work any better for events that are properly read, but contains an unconfigured index as the field value.

Unfortunately, I don't think there is any way to create any 'if-exist' logic at that point....

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.