topic Re: Routing to index based on Regex extraction in Getting Data In

Routing to index based on Regex extraction

Tim_1 — Tue, 19 Sep 2017 14:25:03 GMT

Hi all,

I want to know if it is possible to route data to different indexes based on the value of a regex dynamically.

Example data:

Department:Sec Team, Value=3, Date=12/12/2009
Department:Sales, Value=1, Date=12/03/2010
Department:Other, Value=23, Date=03/02/2011

I know you can hard code the routing like such in transforms.conf:

[route1]
REGEX = "Department:Sec Team"
DEST KEY = _MetaData:Index
FORMAT = index_sec

[route2]
REGEX = "Department:Sales"
DEST KEY = _MetaData:Index
FORMAT = index_sales

[route3]
REGEX = "Department:Other"
DEST KEY = _MetaData:Index
FORMAT = index_other

However, this can become very messy as more and more departments are created (for example).
Is it possible to do something like such?

[route]
REGEX = "Department:<value>"
DEST KEY = _MetaData:Index
FORMAT = index_<value>

I am using Splunk Enterprise 6.4.2

Re: Routing to index based on Regex extraction

alemarzu — Tue, 19 Sep 2017 15:03:18 GMT

Hi there Tim, Im afraid that this is not possible.

Re: Routing to index based on Regex extraction

somesoni2 — Tue, 19 Sep 2017 15:04:52 GMT

First, just want to confirm if there is typo in _Metadata:Index as it should be _MetaData:Index. Second, you can have captured group in REGEX used in FORAMT like this

[route]
 REGEX = "Department:(\S+)"
 DEST KEY = _MetaData:Index
 FORMAT = index_$1

Since there are restrictions in names of index (no spaces or special characters), make sure your capturing group regex is not capturing any of that.

Re: Routing to index based on Regex extraction

Tim_1 — Tue, 19 Sep 2017 15:12:29 GMT

Hi @somesoni2, yes, just a typo when typing the question. Have edited question.
Is it possible to remove spaces in the capture group?
I shall give this a try.

Re: Routing to index based on Regex extraction

cpetterborg — Tue, 19 Sep 2017 21:06:12 GMT

Somesoni2's answer will set the index name to have the uppercase department name (e.g. index_Sales instead of index_sales). Are you needing to use the lowercase? I'm not sure if that can be done. The EVAL parameter of props.conf is where I would do that, and it comes after the TRANSFORMS stuff from the transforms.conf in the index data pipeline, so you could not use that method to lowercase the department. The same may not be said for the removal of spaces in the department name. You can use a SEDCMD in props.conf and then do the assignment of the index in the transforms.conf, but I believe that your _raw data would change. So you could do something like index_SecTeam. You could also do it in the REGEX with something like:

REGEX = "Department:(\S+)\s*(\S*),"
FORMAT = index_$1$2

Make sure you have department indexes all created for data that will directed to the various indexes. You don't want to get lots of error messages saying you are sending to non-existent indexes whenever a new department name shows up in the logs.

Re: Routing to index based on Regex extraction

Tim_1 — Wed, 20 Sep 2017 08:53:54 GMT

@cpetterborg, thanks for the details reply, shall try your suggestions.

Also, I'm assuming that it is impossible to have a dynamic number of capture groups? What I mean by this is that if the value for Department has n number of white spaces, you will need to specify in the REGEX that it has n white spaces.