I want to know if it is possible to route data to different indexes based on the value of a regex dynamically.
Department:Sec Team, Value=3, Date=12/12/2009 Department:Sales, Value=1, Date=12/03/2010 Department:Other, Value=23, Date=03/02/2011
I know you can hard code the routing like such in transforms.conf:
[route1] REGEX = "Department:Sec Team" DEST KEY = _MetaData:Index FORMAT = index_sec [route2] REGEX = "Department:Sales" DEST KEY = _MetaData:Index FORMAT = index_sales [route3] REGEX = "Department:Other" DEST KEY = _MetaData:Index FORMAT = index_other
However, this can become very messy as more and more departments are created (for example).
Is it possible to do something like such?
[route] REGEX = "Department:<value>" DEST KEY = _MetaData:Index FORMAT = index_<value>
I am using Splunk Enterprise 6.4.2
First, just want to confirm if there is typo in
_Metadata:Index as it should be
_MetaData:Index. Second, you can have captured group in REGEX used in FORAMT like this
[route] REGEX = "Department:(\S+)" DEST KEY = _MetaData:Index FORMAT = index_$1
Since there are restrictions in names of index (no spaces or special characters), make sure your capturing group regex is not capturing any of that.
Somesoni2's answer will set the index name to have the uppercase department name (e.g.
index_Sales instead of
index_sales). Are you needing to use the lowercase? I'm not sure if that can be done. The
EVAL parameter of
props.conf is where I would do that, and it comes after the TRANSFORMS stuff from the
transforms.conf in the index data pipeline, so you could not use that method to lowercase the department. The same may not be said for the removal of spaces in the department name. You can use a
props.conf and then do the assignment of the index in the
transforms.conf, but I believe that your
_raw data would change. So you could do something like
index_SecTeam. You could also do it in the REGEX with something like:
REGEX = "Department:(\S+)\s*(\S*)," FORMAT = index_$1$2
Make sure you have department indexes all created for data that will directed to the various indexes. You don't want to get lots of error messages saying you are sending to non-existent indexes whenever a new department name shows up in the logs.
@cpetterborg, thanks for the details reply, shall try your suggestions.
Also, I'm assuming that it is impossible to have a dynamic number of capture groups? What I mean by this is that if the value for Department has
n number of white spaces, you will need to specify in the REGEX that it has
n white spaces.