Splunk Search

Using a transform to set index from event field. How to fall back to a certain index?

Communicator

I have events with a field that contains a desired destination index (see index=* below).

[timestamp] index=layer1 message="123456"
[timestamp] index=layer2 message="123456"
[timestamp] index=layer3 message="123456"
[timestamp] index=jumbled message="123456"
[timestamp] index=layer1 message="123456"
[timestamp] index=layer1 message="123456"
[timestamp] index=layer2 message="123456"

I'm currently using a transform like the following, and it works quite well to get events into their correct index.

[MyTransform]
REGEX = .*index="(.*?)"
DEST_KEY = _MetaData:Index
FORMAT = $1

At least most of the time. Occasionally an event arrives whose index field names an index which doesn't exist (either because the name is wrong or I forgot to create the index). In this case the above transform appears to simply generate an error in the internal index and otherwise drop the data.

Is there another way to do this that would drop messages which failed match an existing index into some catch all index of my choosing?

Something like FORMAT if ($1).exists $1 else myDefaultCatchAllIndex

Tags (3)

Esteemed Legend

You need lastChanceIndex but this is a global setting; this is as close as you can get:
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Indexesconf#GLOBAL_SETTINGS

Esteemed Legend

I would use INGEST_EVAL to do this:

INGEST_EVAL = index = replace(_raw, some stuff here, "")
0 Karma

Explorer

How does that solve the problem of evaluating if an index exists and, if not, fallback to a default index?

0 Karma

Esteemed Legend

OH, I see what you mean. See my new answer.

0 Karma

Splunk Employee
Splunk Employee

You can create another transforms.conf stanza that captures everything and call it in order.

transforms.conf

[MyTransform] 
REGEX = .index="(.?)" 
DEST_KEY = _MetaData:Index 
FORMAT = $1

[MyTransform_all] 
REGEX = . 
DEST_KEY = _MetaData:Index 
FORMAT = catchAll

props.conf

[mysourcetype]
TRANSFORMS-index_routing = MyTransform_all, MyTransform

Explorer

6 years later, is this still not possible?

0 Karma

Splunk Employee
Splunk Employee

You cannot simply define an index on the fly, if that's what you're asking.

0 Karma

Ultra Champion

Yes it matters a lot.

essentially, the TRANSFORMS directive in props.conf sends all of your events on a cruise to two magic islands before going on to Index heaven; On the first island, any event will have it's destination index set to 'catchAll', then it leaves for the second island. There some leprechauns will inspect them and perform some regex magic rites, and only those that have a clear knowledge of where they want to go, will be allowed to do so, i.e. the leprechauns will set their index to the $1 capture group.

When the events come back to dry land, they go to the configured index.

Communicator

does the order of listing them matter? should it be

[mysourcetype] TRANSFORMS-index_routing = MyTransform, MyTransform_all

0 Karma

Ultra Champion

This would work for garbled events, where no match can be made for 'index=someindex', but it will not work any better for events that are properly read, but contains an unconfigured index as the field value.

Unfortunately, I don't think there is any way to create any 'if-exist' logic at that point....

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!