Splunk Search

Certain REGEX strings in transforms.conf will fail

_d_
Splunk Employee
Splunk Employee

I have requierement where i need to route data from certain sources to a specific index. The index name will be extracted from a specific location in the path when it's only preceded by a certain other segment. Example (this is a windows box - notice the path separators):

[send_to_index]
REGEX = bar\\[[:alnum::]]\\
DEST_KEY = queue
FORMAT = indexQueue

In this case I need to extract the path segment that comes after bar and send that path's data to the index with extract's name. The setup is simple and should work but it doesn't.

1 Solution

_d_
Splunk Employee
Splunk Employee

After some careful exploration I have an answer that might benefit the community. The short answer is "do not end your REGEX in backslash".
A btool --debug for the above gives the following:

splunk cmd btool --debug transforms list send_to_index
system [send_to_index]
system CAN_OPTIMIZE = True
system CLEAN_KEYS = True
system DEFAULT_VALUE =
system DEST_KEY =
system FORMAT = indexQueue
system KEEP_EMPTY_VALS = False
system LOOKAHEAD = 4096
system MV_ADD = False
system REGEX = bar\\[[:alnum:]]\
DEST_KEY = queue
system SOURCE_KEY = _raw
system WRITE_META = False

You clearly can see that DEST_KEY is not considered by the system and consequently the setup fails. The culprit behind this appears to be that last backslash.
It turns out that in .conf files the presence of a backslash at the end of a line will escape the implied newline there.

Easy Fix - do NOT end the REGEX with backslash:

[send_to_index]
REGEX = bar\\[[:alnum::]]\\\w+
DEST_KEY = queue
FORMAT = indexQueue

OR, if you need a backslash there, escape it and bracket it.

[send_to_index]
REGEX = bar\\[[:alnum::]][\\]
DEST_KEY = queue
FORMAT = indexQueue

Hope this helps.

View solution in original post

_d_
Splunk Employee
Splunk Employee

After some careful exploration I have an answer that might benefit the community. The short answer is "do not end your REGEX in backslash".
A btool --debug for the above gives the following:

splunk cmd btool --debug transforms list send_to_index
system [send_to_index]
system CAN_OPTIMIZE = True
system CLEAN_KEYS = True
system DEFAULT_VALUE =
system DEST_KEY =
system FORMAT = indexQueue
system KEEP_EMPTY_VALS = False
system LOOKAHEAD = 4096
system MV_ADD = False
system REGEX = bar\\[[:alnum:]]\
DEST_KEY = queue
system SOURCE_KEY = _raw
system WRITE_META = False

You clearly can see that DEST_KEY is not considered by the system and consequently the setup fails. The culprit behind this appears to be that last backslash.
It turns out that in .conf files the presence of a backslash at the end of a line will escape the implied newline there.

Easy Fix - do NOT end the REGEX with backslash:

[send_to_index]
REGEX = bar\\[[:alnum::]]\\\w+
DEST_KEY = queue
FORMAT = indexQueue

OR, if you need a backslash there, escape it and bracket it.

[send_to_index]
REGEX = bar\\[[:alnum::]][\\]
DEST_KEY = queue
FORMAT = indexQueue

Hope this helps.

Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

[Puzzles] Solve, Learn, Repeat: Character substitutions with Regular Expressions

This challenge was first posted on Slack #puzzles channelFor BORE at .conf23, we had a puzzle question which ...

Splunk Community Badges!

  Hey everyone! Ready to earn some serious bragging rights in the community? Along with our existing badges ...

[Puzzles] Solve, Learn, Repeat: Matching cron expressions

This puzzle (first published here) is based on matching timestamps to cron expressions.All the timestamps ...