topic Re: Create index when forward event in Splunk Search

Create index when forward event

stefanosnadal — Mon, 30 Jul 2018 12:29:54 GMT

Is it possible to create index when forward event to the indexer, by extracting value of the field. And this value to be the name of the index?

Re: Create index when forward event

sudosplunk — Mon, 30 Jul 2018 17:27:22 GMT

If you know the value of the field beforehand, using props.conf and transforms.conf, you can achieve this. Please provide some sample data to perform regex matching and your use case with examples. Otherwise, below is the basic structure of configuration settings for routing events.

Props.conf:

[your_custom_sourcetype]
TRANSFORMS-routing = routing_based_on_field_values

Transforms.conf:

[routing_based_on_field_values]
REGEX = <your_custom_regex>
DEST_KEY = _MetaData:Index
FORMAT = <field_value_for_index_name>

You can find more information in below links, let me know if this helps.

http://docs.splunk.com/Documentation/Splunk/7.1.2/Indexer/Setupmultipleindexes#Route_specific_events_to_a_different_index

https://answers.splunk.com/answers/566448/route-specific-events-to-a-relative-index.html

Re: Create index when forward event

stefanosnadal — Tue, 31 Jul 2018 05:40:38 GMT

I'm using this approach, but my idea is to create an index before forwarding. In my case the value of the index is part of the message , but it isn't known and I want to create an index with the same value from the message if it doesn't exist.

Re: Create index when forward event

sudosplunk — Tue, 31 Jul 2018 13:35:11 GMT

I think we can try modifying transforms.conf a little and see if it works. Can you provide some sample events and tell me what value should be extracted.

Re: Create index when forward event

stefanosnadal — Tue, 31 Jul 2018 13:46:57 GMT

Example messages:
{ "name":"value1"} ,
{"name":"value2"},
....
{"name":"valueN"}

The main issue: Is it possible splunk automatically to create indexes with name "value1","value2",... if they does not exist and after that forward messages to that index.

Re: Create index when forward event

FrankVl — Tue, 31 Jul 2018 13:53:55 GMT

No, I don't believe Splunk has the functionality to create a 'missing' index on the fly. Of course you can script anything, so you can also create a saved search that triggers on "received event for unknown index X" errors in the internal logs, which kicks off a script that creates index X. But then at least the first few events will not end up in that index.

Re: Create index when forward event

stefanosnadal — Tue, 31 Jul 2018 14:00:10 GMT

Okey, thanks for the help.

Re: Create index when forward event

sudosplunk — Tue, 31 Jul 2018 14:03:12 GMT

Give this a try. I did not test it.

[routing_based_on_field_values]
REGEX = \{\s?\"(name)\"\:\"(\w+)\"\}
DEST_KEY = _MetaData:Index
FORMAT = $2

Re: Create index when forward event

stefanosnadal — Tue, 31 Jul 2018 14:07:58 GMT

This is used if you have already created the index and you want to forward message to that index.

Re: Create index when forward event

sudosplunk — Tue, 31 Jul 2018 14:28:13 GMT

Actually, the value in the 2nd ($2) capturing group in REGEX is assigned as index value because, DEST_KEY specifies where Splunk stores the expanded FORMAT results in accordance with the REGEX match. Be sure to push these changes to UF, HF (if any) and indexers.

Re: Create index when forward event

FrankVl — Tue, 31 Jul 2018 14:52:25 GMT

That still does not help him for the case where that index doesn't exist yet. That's his whole point. How to dynamically create a new index based on event content.

Re: Create index when forward event

sudosplunk — Tue, 31 Jul 2018 16:22:28 GMT

Agreed! That's why I started my answer with "If you know the value of the field beforehand". But I wanted to see if there are any other possible solutions before giving up. Thank you.