Am I using modular regular expressions wrong?

jeffland — Mon, 06 May 2019 09:04:34 GMT

Hey,

I need to route my data to a different index and append something to the host field if a certain regex matches, following the well know method using props.conf and transforms.conf, for example documented here but also mentioned in transforms.conf. My transforms.conf looks like this (props.conf has TRANSFORMS-class = route_host_by_foo,route_index_by_foo applying this to the appropriate data):

[route_host_by_foo]
REGEX = foo
DEST_KEY = MetaData:Host
FORMAT = $0_custom_suffix
# $0 already contains "host::", so no need to prepend

[route_index_by_foo]
REGEX = foo
DEST_KEY = _MetaData:Index
FORMAT = custom_index

This is working fine. Since I need to change two DEST_KEYS, host and index, and this requires using two transforms.conf stanzas, I've tried to move my regex to a modular regular expression as documented here (search for "MODULAR REGULAR EXPRESSION") to avoid redundant config. It looks like this:

[foo]
REGEX = foo

[route_host_by_foo]
REGEX = [[foo]]
DEST_KEY = MetaData:Host
FORMAT = $0_custom_suffix
# $0 already contains "host::", so no need to prepend

[route_index_by_foo]
REGEX = [[foo]]
DEST_KEY = _MetaData:Index
FORMAT = custom_index

Unfortunately, this doesn't work (same setup as before with props.conf), and I don't see why. Can someone explain?

Update: To clarify, I'm obviously using a more complex regex than foo. Here is my transforms:

# modular regexes
[app_1_schema]
REGEX = (?:regex_1|regex_2|...)

[app_1_user]
REGEX = (?:regex_3|regex_4|...)

[app_2_schema]
REGEX = (?:regex_5|regex_6|...)

[app_2_user]
REGEX = (?:regex_7|regex_8|...)

# actual props.conf mentioned transforms
[app_1_rewrite_index_by_user]
REGEX = <DB_User>([[app_1_schema]]|[[app_1_user]])<\/DB_User>
DEST_KEY = _MetaData:Index
FORMAT = app_2

[app_2_rewrite_index_by_user]
REGEX = <DB_User>([[app_2_schema]]|[[app_2_user]])<\/DB_User>
DEST_KEY = _MetaData:Index
FORMAT = app_1

[app_1_rewrite_host_by_user]
REGEX = <DB_User>([[app_1_schema]]|[[app_1_user]])<\/DB_User>
DEST_KEY = MetaData:Host
FORMAT = $0_app_2

[app_2_rewrite_host_by_user]
REGEX = <DB_User>([[app_2_schema]]|[[app_2_user]])<\/DB_User>
DEST_KEY = MetaData:Host
FORMAT = $0_app_1

[app_1_rewrite_index_by_schema]
REGEX = <Object_Schema>[[app_1_schema]]<\/Object_Schema>
DEST_KEY = _MetaData:Index
FORMAT = app_2

[app_2_rewrite_index_by_schema]
REGEX = <Object_Schema>[[app_2_schema]]<\/Object_Schema>
DEST_KEY = _MetaData:Index
FORMAT = app_1

[app_1_rewrite_host_by_schema]
REGEX = Object_Schema>[[app_1_schema]]<\/Object_Schema>
DEST_KEY = MetaData:Host
FORMAT = $0_app_2

[app_2_rewrite_host_by_schema]
REGEX = <Object_Schema>[[app_2_schema]]<\/Object_Schema>
DEST_KEY = MetaData:Host
FORMAT = $0_app_1

This does not work. 's/\[\[app_1_schema]]/(?:regex_1|regex_2|...)/g' etc. makes it work.
Also, the events are not simply left untouched - they disappear. I have an index and host set up at the input level (on a forwarder), and there are events that do not match the above regexes. Those continue into their expected index regardless of whether I'm using modular regular expressions or not. The ones where the regex matches are either successfully routed to the new index with a new host as long as I'm using the explicit REGEX in the transforms.conf stanza mentioned by props.conf, or they disappear when I use modular regular expressions.

Re: Am I using modular regular expressions wrong?

493669 — Mon, 06 May 2019 09:20:28 GMT

Hi @jeffland,
Try to use below

DEST_KEY = MetaData:Host

Since Keys are case-sensitive.

Re: Am I using modular regular expressions wrong?

jeffland — Mon, 06 May 2019 09:35:44 GMT

That was a typo in my question (now corrected), the config works as expected when not using modular regular expressions.

Re: Am I using modular regular expressions wrong?

woodcock — Mon, 06 May 2019 17:37:32 GMT

That seems like it should work to me; are you sure that it doesn't? For an example of a working app, you can see how this one does it:
https://splunkbase.splunk.com/app/3121/

Re: Am I using modular regular expressions wrong?

jeffland — Tue, 07 May 2019 07:48:05 GMT

Yes, I'm sure. I'll update the answer with more specific code of what I'm doing. Thanks for hinting me at that app, however it strictly uses it with transforms field extractions which is exactly what the docs also do (unlike my example, which uses it for event routing).

Re: Am I using modular regular expressions wrong?

woodcock — Sat, 11 May 2019 19:09:08 GMT

I would open a support case.

topic Re: Am I using modular regular expressions wrong? in Getting Data In

Am I using modular regular expressions wrong?

Re: Am I using modular regular expressions wrong?

Re: Am I using modular regular expressions wrong?

Re: Am I using modular regular expressions wrong?

Re: Am I using modular regular expressions wrong?

Re: Am I using modular regular expressions wrong?