How to replace all "confusable" characters in field in data model?

[jasmartin]

Hello, I am attempting to replace a large unwieldy macro with a data model. Part of the macro is a rex command that finds what we call "confusable characters" that are the highbit versions of ASCII characters, like 𝟐 or ꓜ, and replaces them with the ASCII versions (2 or Z respectively), like this:

rex field=$arg1$ mode=sed "y/𝟐𝟚𝟤𝟮𝟸ꝚƧϨꙄᒿꛯ/22222222222/"

The actual macro is much longer and encompasses all numbers and letters.

I have been having difficultly figuring out how to incorporate this into the data model. I've been able to use a CSV lookup like this:

char_search,old_char,new_char
*𝟐*,𝟐,2
*ꓜ*,ꓜ,Z

Make char_search a wildcard match field, and use this query:

| makeresults
| eval t="dfasdf𝟐𝟐"
| lookup CSVconfusables char_search as t OUTPUT
| eval u=replace(t,old_char,new_char)

It works find with 1 character to replace, but when there are multiple to replace, the lookup output fields become multivalue and replace doesn't work:

| makeresults
| eval t="ꓜdfasdf𝟐𝟐"
| lookup CSVconfusables char_search as t OUTPUT
| eval u=replace(t,old_char,new_char)

Is there any way to accomplish what the macro is doing in a data model? Thanks in advance!

[richgalloway]

Since the replace function uses regular expressions as does rex, have you tried putting the expressions from the macro into the DM?

| eval arg1=replace(arg1,"𝟐𝟚𝟤𝟮𝟸ꝚƧϨꙄᒿꛯ","22222222222")

[jasmartin]

Hi richgalloway , unfortunately that doesn't do what I need it to do. replace searches for the whole given string to replace. I want to do what sed does with "y/", where it's a one-on-one replacement.

[PickleRick · Accepted solution]

@richgalloway's idea was close. If I understand you well, you need something like

| eval arg1=replace(arg1,"[𝟐𝟚𝟤𝟮𝟸ꝚƧϨꙄᒿꛯ]","2")

[jasmartin]

That will work, thank you!