Getting Data In

Routing to index based on Regex extraction

Path Finder

Hi all,

I want to know if it is possible to route data to different indexes based on the value of a regex dynamically.

Example data:

Department:Sec Team, Value=3, Date=12/12/2009
Department:Sales, Value=1, Date=12/03/2010
Department:Other, Value=23, Date=03/02/2011

I know you can hard code the routing like such in transforms.conf:

REGEX = "Department:Sec Team"
DEST KEY = _MetaData:Index
FORMAT = index_sec

REGEX = "Department:Sales"
DEST KEY = _MetaData:Index
FORMAT = index_sales

REGEX = "Department:Other"
DEST KEY = _MetaData:Index
FORMAT = index_other

However, this can become very messy as more and more departments are created (for example).
Is it possible to do something like such?

REGEX = "Department:<value>"
DEST KEY = _MetaData:Index
FORMAT = index_<value>

I am using Splunk Enterprise 6.4.2

0 Karma

Revered Legend

First, just want to confirm if there is typo in _Metadata:Index as it should be _MetaData:Index. Second, you can have captured group in REGEX used in FORAMT like this

 REGEX = "Department:(\S+)"
 DEST KEY = _MetaData:Index
 FORMAT = index_$1

Since there are restrictions in names of index (no spaces or special characters), make sure your capturing group regex is not capturing any of that.

0 Karma

Path Finder

Hi @somesoni2, yes, just a typo when typing the question. Have edited question.
Is it possible to remove spaces in the capture group?
I shall give this a try.

0 Karma


Somesoni2's answer will set the index name to have the uppercase department name (e.g. index_Sales instead of index_sales). Are you needing to use the lowercase? I'm not sure if that can be done. The EVAL parameter of props.conf is where I would do that, and it comes after the TRANSFORMS stuff from the transforms.conf in the index data pipeline, so you could not use that method to lowercase the department. The same may not be said for the removal of spaces in the department name. You can use a SEDCMD in props.conf and then do the assignment of the index in the transforms.conf, but I believe that your _raw data would change. So you could do something like index_SecTeam. You could also do it in the REGEX with something like:

REGEX = "Department:(\S+)\s*(\S*),"
FORMAT = index_$1$2

Make sure you have department indexes all created for data that will directed to the various indexes. You don't want to get lots of error messages saying you are sending to non-existent indexes whenever a new department name shows up in the logs.

0 Karma

Path Finder

@cpetterborg, thanks for the details reply, shall try your suggestions.

Also, I'm assuming that it is impossible to have a dynamic number of capture groups? What I mean by this is that if the value for Department has n number of white spaces, you will need to specify in the REGEX that it has n white spaces.

0 Karma


Hi there Tim, Im afraid that this is not possible.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!