Create index when forward event

[stefanosnadal]

Is it possible to create index when forward event to the indexer, by extracting value of the field. And this value to be the name of the index?

[FrankVl]

No, I don't believe Splunk has the functionality to create a 'missing' index on the fly. Of course you can script anything, so you can also create a saved search that triggers on "received event for unknown index X" errors in the internal logs, which kicks off a script that creates index X. But then at least the first few events will not end up in that index.

[stefanosnadal]

Okey, thanks for the help.

[sudosplunk]

If you know the value of the field beforehand, using props.conf and transforms.conf, you can achieve this. Please provide some sample data to perform regex matching and your use case with examples. Otherwise, below is the basic structure of configuration settings for routing events.

Props.conf:

[your_custom_sourcetype]
TRANSFORMS-routing = routing_based_on_field_values

Transforms.conf:

[routing_based_on_field_values]
REGEX = <your_custom_regex>
DEST_KEY = _MetaData:Index
FORMAT = <field_value_for_index_name>

You can find more information in below links, let me know if this helps.

http://docs.splunk.com/Documentation/Splunk/7.1.2/Indexer/Setupmultipleindexes#Route_specific_events...

https://answers.splunk.com/answers/566448/route-specific-events-to-a-relative-index.html

[stefanosnadal]

I'm using this approach, but my idea is to create an index before forwarding. In my case the value of the index is part of the message , but it isn't known and I want to create an index with the same value from the message if it doesn't exist.

[sudosplunk]

I think we can try modifying transforms.conf a little and see if it works. Can you provide some sample events and tell me what value should be extracted.

[stefanosnadal]

Example messages:
{ "name":"value1"} ,
{"name":"value2"},
....
{"name":"valueN"}

The main issue: Is it possible splunk automatically to create indexes with name "value1","value2",... if they does not exist and after that forward messages to that index.

[sudosplunk]

Give this a try. I did not test it.

[routing_based_on_field_values]
REGEX = \{\s?\"(name)\"\:\"(\w+)\"\}
DEST_KEY = _MetaData:Index
FORMAT = $2

[stefanosnadal]

This is used if you have already created the index and you want to forward message to that index.

[sudosplunk]

Actually, the value in the 2nd ($2) capturing group in REGEX is assigned as index value because, DEST_KEY specifies where Splunk stores the expanded FORMAT results in accordance with the REGEX match. Be sure to push these changes to UF, HF (if any) and indexers.

[FrankVl]

That still does not help him for the case where that index doesn't exist yet. That's his whole point. How to dynamically create a new index based on event content.

[sudosplunk]

Agreed! That's why I started my answer with "If you know the value of the field beforehand". But I wanted to see if there are any other possible solutions before giving up. Thank you.